Jul 23 17:58:50.168840 ip-10-0-129-67 systemd[1]: Starting Kubernetes Kubelet... Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:58:50.629533 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.623378 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630710 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630724 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630728 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630732 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630735 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630738 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630742 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630745 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630748 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630751 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630754 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630756 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630759 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630763 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630765 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630768 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630771 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630780 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630782 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:50.727829 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630785 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:50.709952 ip-10-0-129-67 systemd[1]: Started Kubernetes Kubelet. Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630788 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630791 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630793 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630796 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630800 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630805 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630808 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630811 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630813 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630816 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630818 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630821 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630824 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630827 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630829 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630832 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630834 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630837 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:50.748285 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630841 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630846 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630848 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630851 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630854 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630856 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630859 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630861 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630864 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630866 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630869 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630871 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630874 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630877 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630880 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630885 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630889 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630892 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630895 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:50.963340 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630898 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630901 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630904 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630907 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630910 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630913 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630915 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630918 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630920 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630923 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630926 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630929 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630932 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630934 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630939 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630943 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630945 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630949 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630952 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:50.973538 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630954 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630957 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630961 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630964 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630966 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630969 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630972 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630975 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630978 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630981 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630983 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630986 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630988 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630991 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630994 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630996 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.630999 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631002 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631004 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631007 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631009 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:50.974988 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631012 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.631014 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633086 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633098 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633101 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633106 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633109 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633112 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633115 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633118 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633120 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633124 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633126 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633129 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633132 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633134 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633138 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633140 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633143 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633146 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:50.975956 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633148 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633151 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633154 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633157 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633160 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633162 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633165 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633168 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633170 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633175 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633180 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633183 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633186 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633190 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633193 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633196 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633199 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633202 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633204 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:50.977039 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633207 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633210 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633212 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633215 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633218 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633222 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633225 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633228 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633231 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633234 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633237 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633240 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633243 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633245 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633248 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633250 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633253 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633256 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633258 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:50.977928 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633261 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633263 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633266 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633271 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633274 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633277 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633280 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633282 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633286 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633289 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633291 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633294 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633297 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633299 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633301 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633304 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633307 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633309 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633313 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633316 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:50.979036 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633319 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633322 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633325 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633328 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633330 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633333 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633335 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633339 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633341 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633344 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633347 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633349 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633352 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633355 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633357 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633361 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633363 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633366 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633369 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633372 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:50.980140 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633374 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633377 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633379 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633824 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633830 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633834 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633837 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633840 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633843 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633845 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633850 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633853 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633856 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633859 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633862 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633866 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633868 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633871 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633874 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:50.981019 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633877 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633880 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633882 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633885 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633888 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633890 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633893 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633896 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633900 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633902 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633905 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633909 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633912 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633914 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633917 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633919 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633922 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633925 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633927 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633930 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:50.982103 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633932 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633935 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633937 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633940 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633942 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633945 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633947 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633950 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633952 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633955 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633957 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633960 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633962 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633965 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633968 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633970 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633973 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633975 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633978 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633980 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:50.982983 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633985 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633989 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633991 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633995 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.633998 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634000 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634003 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634006 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634008 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634010 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634013 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634016 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634018 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634021 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634023 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634025 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634029 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634031 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634034 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634037 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:50.984060 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634040 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634044 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634047 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634050 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634053 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634057 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634060 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634063 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634066 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634069 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634071 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634074 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634078 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634081 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634084 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634087 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634090 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634093 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634095 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:50.984922 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634098 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634100 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634103 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634105 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634171 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634178 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634184 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634188 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634192 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634195 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634200 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634205 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634208 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634212 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634215 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634219 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634221 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634224 2568 flags.go:64] FLAG: --cgroup-root="" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634227 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634230 2568 flags.go:64] FLAG: --client-ca-file="" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634234 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634237 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634245 2568 flags.go:64] FLAG: --cluster-domain="" Jul 23 17:58:50.985791 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634248 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634251 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634256 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634260 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634268 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634273 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634276 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634279 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634282 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634285 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634288 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634291 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634294 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634298 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634301 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634304 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634307 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634310 2568 flags.go:64] FLAG: --enable-server="true" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634312 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634317 2568 flags.go:64] FLAG: --event-burst="100" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634320 2568 flags.go:64] FLAG: --event-qps="50" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634322 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634325 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634328 2568 flags.go:64] FLAG: --eviction-hard="" Jul 23 17:58:50.986874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634332 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634335 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634338 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634341 2568 flags.go:64] FLAG: --eviction-soft="" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634345 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634347 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634351 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634354 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634357 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634359 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634363 2568 flags.go:64] FLAG: --feature-gates="" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634367 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634370 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634374 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634378 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634380 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634383 2568 flags.go:64] FLAG: --help="false" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634386 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-67.ec2.internal" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634389 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634393 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634395 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634399 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634402 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634405 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 17:58:50.987948 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634408 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634411 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634414 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634417 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634420 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634422 2568 flags.go:64] FLAG: --kube-reserved="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634425 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634428 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634431 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634434 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634437 2568 flags.go:64] FLAG: --lock-file="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634440 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634443 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634446 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634451 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634454 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634457 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634460 2568 flags.go:64] FLAG: --logging-format="text" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634464 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634467 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634470 2568 flags.go:64] FLAG: --manifest-url="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634474 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634478 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634481 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634485 2568 flags.go:64] FLAG: --max-pods="110" Jul 23 17:58:50.989035 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634488 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634491 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634494 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634497 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634500 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634503 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634506 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634513 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634516 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634519 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634522 2568 flags.go:64] FLAG: --pod-cidr="" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634525 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634530 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634533 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634536 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634539 2568 flags.go:64] FLAG: --port="10250" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634542 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634545 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-002aa7d414f36455d" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634548 2568 flags.go:64] FLAG: --qos-reserved="" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634551 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634568 2568 flags.go:64] FLAG: --register-node="true" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634572 2568 flags.go:64] FLAG: --register-with-taints="" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634576 2568 flags.go:64] FLAG: --registry-burst="10" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634580 2568 flags.go:64] FLAG: --registry-qps="5" Jul 23 17:58:50.990118 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634582 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634591 2568 flags.go:64] FLAG: --reserved-memory="" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634598 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634601 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634607 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634610 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634613 2568 flags.go:64] FLAG: --runonce="false" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634616 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634619 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634622 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634625 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634628 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634631 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634634 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634637 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634640 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634643 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634645 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634648 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634651 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634654 2568 flags.go:64] FLAG: --system-cgroups="" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634657 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634662 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634665 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634668 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 17:58:50.991196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634674 2568 flags.go:64] FLAG: --tls-min-version="" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634676 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634679 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634682 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634684 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634688 2568 flags.go:64] FLAG: --v="2" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634692 2568 flags.go:64] FLAG: --version="false" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634701 2568 flags.go:64] FLAG: --vmodule="" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634705 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.634710 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634832 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634838 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634841 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634844 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634847 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634849 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634853 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634855 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634858 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634861 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634863 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:50.992287 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634866 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634869 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634871 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634874 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634877 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634879 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634883 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634888 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634892 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634895 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634898 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634900 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634903 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634906 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634909 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634912 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634915 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634917 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634920 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:50.995282 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634923 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634927 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634930 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634934 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634936 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634939 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634941 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634944 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634947 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634949 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634952 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634955 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634957 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634960 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634962 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634965 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634967 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634970 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634973 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634977 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:50.995898 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634980 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634983 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634985 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634988 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634990 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634993 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634995 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.634998 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635000 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635003 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635005 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635008 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635010 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635015 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635017 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635021 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635024 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635027 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635029 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635032 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:51.001326 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635034 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635037 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635039 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635042 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635045 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635047 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635049 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635052 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635054 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635057 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635059 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635062 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635065 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635067 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635070 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635073 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635075 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635077 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635080 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635082 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635085 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:51.001937 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635087 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635090 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635092 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635095 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635099 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635103 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635107 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.635109 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.635846 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.643051 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.643071 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.643092 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.643098 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643169 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643176 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:51.002487 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643180 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643184 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643188 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643191 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643193 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643197 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643200 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643203 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643205 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643209 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643212 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643214 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643217 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643220 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643226 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643229 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643234 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643238 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643241 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643245 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:51.141086 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643248 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643251 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643257 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643261 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643264 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643268 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643272 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643277 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643281 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643284 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643287 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643290 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643293 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643297 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643300 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643302 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643305 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643307 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643310 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:51.141822 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643314 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643320 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643323 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643328 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643331 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643334 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643337 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643342 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643345 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643348 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643351 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643354 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643359 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643362 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643366 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643369 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643372 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643375 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643378 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:51.142432 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643381 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643383 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643388 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643390 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643393 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643396 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643400 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643402 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643405 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643407 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643411 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643414 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643417 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643420 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643423 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643427 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643429 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643432 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643435 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643439 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:51.143063 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643443 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643446 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643450 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643453 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643456 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643461 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643464 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643466 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643469 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643472 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643475 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643478 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643480 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643484 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643487 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643490 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643493 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643497 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:51.143694 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643500 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.643507 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643686 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643693 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643697 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643701 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643703 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643706 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643709 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643713 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643715 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643719 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643722 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643725 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643728 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:58:51.144312 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643732 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643734 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643737 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643741 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643749 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643752 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643756 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643758 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643761 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643765 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643768 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643771 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643773 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643777 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643780 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643784 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643787 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643790 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643793 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643796 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:58:51.144931 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643798 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643802 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643804 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643807 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643809 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643813 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643816 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643819 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643822 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643825 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643828 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643831 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643836 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643840 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643843 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643846 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643850 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643853 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643856 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643858 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:58:51.145549 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643862 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643865 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643868 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643870 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643874 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643877 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643880 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643883 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643887 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643890 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643892 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643895 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643899 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643903 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643906 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643909 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643913 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643916 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643918 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:58:51.146113 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643921 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643925 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643928 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643931 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643935 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643940 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643942 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643945 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643947 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643951 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643955 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643958 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643961 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643964 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643967 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643970 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643974 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643980 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:58:51.146645 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643986 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643989 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643993 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.643996 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.644007 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.644010 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.644013 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.644016 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:50.644019 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.644023 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.644174 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.646684 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.648327 2568 server.go:1019] "Starting client certificate rotation" Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.648432 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:58:51.147159 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.648471 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.651289 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.657649 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.671177 2568 log.go:25] "Validated CRI v1 runtime API" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.673643 2568 log.go:25] "Validated CRI v1 image API" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.674401 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.677068 2568 fs.go:135] Filesystem UUIDs: map[0f79b246-93d7-4495-a9bb-8375442fb72e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9dc9b216-fb8e-4136-b5c2-0545611c3a08:/dev/nvme0n1p4] Jul 23 17:58:51.147546 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.677085 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.682923 2568 manager.go:216] Machine: {Timestamp:2026-07-23 17:58:50.681373406 +0000 UTC m=+0.372163429 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100022 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20ee195c1f6c14df675da99913fda9 SystemUUID:ec20ee19-5c1f-6c14-df67-5da99913fda9 BootID:6d9fad4b-539d-40c8-bab5-bc86dda24cf0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:bc:b2:fb:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:bc:b2:fb:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:5b:d9:c8:f1:6c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.683031 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.683116 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.685312 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.687096 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.687125 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.688110 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.688120 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.688127 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.688143 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.690454 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694051 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694198 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694225 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694234 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694246 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 17:58:51.147851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694255 2568 kubelet.go:398] "Adding apiserver pod source" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694264 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694713 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.694737 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.697168 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.699057 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.699084 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700575 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700590 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700596 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700602 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700610 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700616 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700624 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700631 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700637 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700650 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700666 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700672 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700693 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.700698 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.704184 2568 server.go:1268] "Started kubelet" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.704231 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.704292 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.704362 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.708722 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.713057 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.713033 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.713039 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.713136 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.713205 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.714808 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18c4fd00723c4be2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-07-23 17:58:50.703940578 +0000 UTC m=+0.394730597,LastTimestamp:2026-07-23 17:58:50.703940578 +0000 UTC m=+0.394730597,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720125 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720194 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.720221 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720884 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720886 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720916 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720969 2568 factory.go:55] Registering systemd factory Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.720986 2568 factory.go:223] Registration of the systemd container factory successfully Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721020 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 23 17:58:51.148379 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721027 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721241 2568 factory.go:153] Registering CRI-O factory Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721251 2568 factory.go:223] Registration of the crio container factory successfully Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721299 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721319 2568 factory.go:103] Registering Raw factory Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721328 2568 manager.go:1196] Started watching for new ooms in manager Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.721650 2568 manager.go:349] Starting recovery of all containers Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.728268 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.728405 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.732247 2568 manager.go:354] Recovery completed Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.736255 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738031 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738059 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738069 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738538 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738545 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738546 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.738576 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.739311 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.739333 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.739346 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.740741 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18c4fd007444ae7a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-07-23 17:58:50.738044538 +0000 UTC m=+0.428834557,LastTimestamp:2026-07-23 17:58:50.738044538 +0000 UTC m=+0.428834557,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.742601 2568 policy_none.go:49] "None policy: Start" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.742615 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.742626 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.748286 2568 policy_none.go:47] "Start" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.751646 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8hnv2" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.752031 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18c4fd007444f9c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-67.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-07-23 17:58:50.738063808 +0000 UTC m=+0.428853827,LastTimestamp:2026-07-23 17:58:50.738063808 +0000 UTC m=+0.428853827,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}" Jul 23 17:58:51.149518 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.761137 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8hnv2" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.764484 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-67.ec2.internal.18c4fd0074451bcd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-67.ec2.internal,UID:ip-10-0-129-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-67.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-67.ec2.internal,},FirstTimestamp:2026-07-23 17:58:50.738072525 +0000 UTC m=+0.428862544,LastTimestamp:2026-07-23 17:58:50.738072525 +0000 UTC m=+0.428862544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-67.ec2.internal,}" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807118 2568 manager.go:337] "Starting Device Plugin manager" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.807150 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807160 2568 server.go:87] "Starting device plugin registration server" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807446 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807457 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807583 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807671 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.807679 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.808208 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.808261 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.829048 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.830263 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.830301 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.830323 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.830358 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.833594 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.907885 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.908856 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.908884 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.908894 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.908916 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.919057 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.919074 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-67.ec2.internal\": node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.930663 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal"] Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.930716 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.931481 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.931509 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.931521 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.932865 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.932963 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.932990 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933495 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933500 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933524 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933532 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.150315 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933535 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.933542 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.935350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.935378 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.936545 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.936584 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:50.936595 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:50.973713 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.009518 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-67.ec2.internal\" not found" node="ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.013738 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-67.ec2.internal\" not found" node="ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.022811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.022833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.022851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100d0bec6eaa8166829fa6f437da8a36-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"100d0bec6eaa8166829fa6f437da8a36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.074805 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.123929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.123954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.123971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100d0bec6eaa8166829fa6f437da8a36-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"100d0bec6eaa8166829fa6f437da8a36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.124013 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100d0bec6eaa8166829fa6f437da8a36-config\") pod \"kube-apiserver-proxy-ip-10-0-129-67.ec2.internal\" (UID: \"100d0bec6eaa8166829fa6f437da8a36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.124044 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.151663 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.124047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2570112e3a53f32ff7db5d034b5370-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal\" (UID: \"4c2570112e3a53f32ff7db5d034b5370\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.218706 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.175845 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.293186 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.276505 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.388821 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.312679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.388821 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:51.316381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:51.388821 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.377577 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.509409 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.478150 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:51.618875 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:51.578612 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:53.026673 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:52.964297 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 17:58:53.026673 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:52.964458 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 17:58:53.026673 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:52.989711 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:58:53.026673 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:52.995670 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:53.129712 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.083657 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 17:53:52 +0000 UTC" deadline="2028-03-12 21:23:07.19414366 +0000 UTC" Jul 23 17:58:53.129712 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.083703 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14355h24m14.110444475s" Jul 23 17:58:53.129712 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:53.095743 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.192714 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.193946 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.193975 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.193985 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:53.196220 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-67.ec2.internal\" not found" Jul 23 17:58:53.243346 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.237880 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:58:53.347985 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:53.312883 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100d0bec6eaa8166829fa6f437da8a36.slice/crio-2da141940d7f862ae87b48cdcece9b6ea6f1569d536f67ec7e6d0a52014181f9 WatchSource:0}: Error finding container 2da141940d7f862ae87b48cdcece9b6ea6f1569d536f67ec7e6d0a52014181f9: Status 404 returned error can't find the container with id 2da141940d7f862ae87b48cdcece9b6ea6f1569d536f67ec7e6d0a52014181f9 Jul 23 17:58:53.347985 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:53.313389 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2570112e3a53f32ff7db5d034b5370.slice/crio-357f496c4c31aad7e0520901d39391f45319663d82b122cb7725406943c6dc52 WatchSource:0}: Error finding container 357f496c4c31aad7e0520901d39391f45319663d82b122cb7725406943c6dc52: Status 404 returned error can't find the container with id 357f496c4c31aad7e0520901d39391f45319663d82b122cb7725406943c6dc52 Jul 23 17:58:53.347985 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.318942 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 17:58:53.347985 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.337215 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" Jul 23 17:58:53.347985 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.337345 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:58:53.490656 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.357185 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:58:53.490656 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.358255 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" Jul 23 17:58:53.490656 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.369762 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:58:53.775833 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:53.775756 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:58:54.011434 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.011400 2568 apiserver.go:52] "Watching apiserver" Jul 23 17:58:54.020398 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.020363 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 17:58:54.021395 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.021371 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jswkv","openshift-image-registry/node-ca-fjkxn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal","openshift-multus/multus-additional-cni-plugins-9gglm","openshift-network-operator/iptables-alerter-dm92m","openshift-ovn-kubernetes/ovnkube-node-g55gm","kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7","openshift-cluster-node-tuning-operator/tuned-dlbwd","openshift-dns/node-resolver-bvbtd","openshift-multus/multus-8bbvd","openshift-multus/network-metrics-daemon-vlpnb","openshift-network-diagnostics/network-check-target-s789k","kube-system/global-pull-secret-syncer-qv8b5"] Jul 23 17:58:54.024515 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.024492 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.026785 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.026677 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:54.027460 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.026780 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:58:54.027460 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.027052 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.027460 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.027057 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.027460 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.027380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gh5p5\"" Jul 23 17:58:54.031044 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.031021 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.031151 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.031083 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:58:54.031494 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.031477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.033675 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.033656 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 17:58:54.033854 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.033835 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 17:58:54.033964 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.033945 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 17:58:54.034233 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.034210 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.035031 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.035009 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.035325 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.035310 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x2d9n\"" Jul 23 17:58:54.035400 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.035328 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.036402 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.036385 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.037028 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.037002 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.037195 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.037176 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jcpl2\"" Jul 23 17:58:54.037324 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.037309 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 17:58:54.037376 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.037363 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 17:58:54.038264 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.038244 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.038917 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.038894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.039047 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.039029 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 17:58:54.039544 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.039528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 17:58:54.040482 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.040462 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.040707 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.040673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.040889 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.040675 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 17:58:54.041371 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.041351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.041538 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.041520 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 17:58:54.042866 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.042849 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kb9x4\"" Jul 23 17:58:54.042954 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.042898 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.042954 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.042854 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.043093 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.043077 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6l4xb\"" Jul 23 17:58:54.043525 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.043509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.043594 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.043580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 17:58:54.044043 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.044026 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.044503 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.044488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rw676\"" Jul 23 17:58:54.044792 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.044772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.047066 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.047048 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.047456 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.047432 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 17:58:54.047541 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.047522 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kddb5\"" Jul 23 17:58:54.047867 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.047849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 17:58:54.047924 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.047884 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 17:58:54.049378 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.049359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.049456 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.049388 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cv5h\"" Jul 23 17:58:54.049616 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.049594 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 17:58:54.050694 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.050661 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.053180 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.053272 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054142 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054478 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7dxcl\"" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-systemd\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-host\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-slash\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-etc-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-config\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054940 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kplm\" (UniqueName: \"kubernetes.io/projected/343419e3-9da3-4649-8286-a105c42fc915-kube-api-access-8kplm\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.054974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-systemd-units\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-bin\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/343419e3-9da3-4649-8286-a105c42fc915-ovn-node-metrics-cert\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-script-lib\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.058719 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-modprobe-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055240 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-run\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bd4m\" (UniqueName: \"kubernetes.io/projected/fb4892e0-666c-462a-a2b0-98fca0d564b4-kube-api-access-5bd4m\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-systemd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysconfig\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-tmp\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-node-log\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055574 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-env-overrides\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4c5948-06e9-4626-8545-e242dde01132-tmp-dir\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-dbus\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-cnibin\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-os-release\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-kubernetes\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.059602 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-sys\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-tuned\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.055968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-system-cni-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-iptables-alerter-script\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-host-slash\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-conf\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-lib-modules\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-var-lib-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-ovn\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056344 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-netd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7xn\" (UniqueName: \"kubernetes.io/projected/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-kube-api-access-8n7xn\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-var-lib-kubelet\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-netns\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.060366 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-log-socket\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056608 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-kubelet-config\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wkg7\" (UniqueName: \"kubernetes.io/projected/d03e58df-c1b1-4de8-916d-b68257dd562e-kube-api-access-2wkg7\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-kubelet\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef4c5948-06e9-4626-8545-e242dde01132-hosts-file\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.061157 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.056824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wfr\" (UniqueName: \"kubernetes.io/projected/ef4c5948-06e9-4626-8545-e242dde01132-kube-api-access-q7wfr\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.061896 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.061875 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:58:54.098939 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.098917 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjdhb" Jul 23 17:58:54.110874 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.110850 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjdhb" Jul 23 17:58:54.138829 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.138800 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 17:58:54.151639 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.151590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"4c2570112e3a53f32ff7db5d034b5370","Type":"ContainerStarted","Data":"357f496c4c31aad7e0520901d39391f45319663d82b122cb7725406943c6dc52"} Jul 23 17:58:54.153062 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.153022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" event={"ID":"100d0bec6eaa8166829fa6f437da8a36","Type":"ContainerStarted","Data":"2da141940d7f862ae87b48cdcece9b6ea6f1569d536f67ec7e6d0a52014181f9"} Jul 23 17:58:54.157369 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wkg7\" (UniqueName: \"kubernetes.io/projected/d03e58df-c1b1-4de8-916d-b68257dd562e-kube-api-access-2wkg7\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-kubelet\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qkb\" (UniqueName: \"kubernetes.io/projected/ae691532-a868-4769-bde5-c93f7eb4e1d0-kube-api-access-69qkb\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-etc-kubernetes\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157491 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-host\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-slash\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157543 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-etc-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-config\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2rl\" (UniqueName: \"kubernetes.io/projected/8436bc20-fc7b-4845-b5df-7b681ee5fee0-kube-api-access-9j2rl\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157638 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-systemd-units\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-bin\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-script-lib\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-socket-dir-parent\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bd4m\" (UniqueName: \"kubernetes.io/projected/fb4892e0-666c-462a-a2b0-98fca0d564b4-kube-api-access-5bd4m\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4c5948-06e9-4626-8545-e242dde01132-tmp-dir\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9447ea52-6b76-4675-a2fd-447b95a0b740-agent-certs\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysconfig\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.157912 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-tmp\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-multus\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157942 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-cnibin\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.157991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-os-release\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-kubernetes\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-tuned\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-netd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158100 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-sys-fs\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-iptables-alerter-script\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-host-slash\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-conf\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-var-lib-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpm6\" (UniqueName: \"kubernetes.io/projected/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kube-api-access-2cpm6\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-netns\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-log-socket\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.158498 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158354 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-device-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9447ea52-6b76-4675-a2fd-447b95a0b740-konnectivity-ca\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-kubelet-config\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158433 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wfr\" (UniqueName: \"kubernetes.io/projected/ef4c5948-06e9-4626-8545-e242dde01132-kube-api-access-q7wfr\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-socket-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cnibin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cni-binary-copy\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-multus-certs\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-systemd\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-kubelet\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kplm\" (UniqueName: \"kubernetes.io/projected/343419e3-9da3-4649-8286-a105c42fc915-kube-api-access-8kplm\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-os-release\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/343419e3-9da3-4649-8286-a105c42fc915-ovn-node-metrics-cert\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-k8s-cni-cncf-io\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-modprobe-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-run\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.159326 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158796 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-systemd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-systemd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bnh\" (UniqueName: \"kubernetes.io/projected/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-kube-api-access-98bnh\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-node-log\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.158999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-env-overrides\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-registration-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae691532-a868-4769-bde5-c93f7eb4e1d0-host\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae691532-a868-4769-bde5-c93f7eb4e1d0-serviceca\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-system-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-dbus\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-sys\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159196 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4c5948-06e9-4626-8545-e242dde01132-tmp-dir\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-ovn\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-bin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysconfig\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-kubelet\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-hostroot\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-system-cni-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-lib-modules\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-netns\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159442 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-conf-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-daemon-config\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7xn\" (UniqueName: \"kubernetes.io/projected/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-kube-api-access-8n7xn\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-var-lib-kubelet\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef4c5948-06e9-4626-8545-e242dde01132-hosts-file\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159585 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.160823 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159786 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-host\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159826 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-slash\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.159864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-etc-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-cnibin\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160076 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-os-release\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-kubernetes\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160449 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-config\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160649 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-systemd-units\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.160700 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-bin\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161067 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-cni-netd\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-ovnkube-script-lib\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.161537 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161496 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-host-slash\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-iptables-alerter-script\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161620 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-sysctl-conf\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161650 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-var-lib-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161699 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161751 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161826 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-node-log\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.161829 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161907 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-host-run-netns\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.161912 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:54.661890967 +0000 UTC m=+3.036554141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.161947 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-log-socket\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162125 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-kubelet-config\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d03e58df-c1b1-4de8-916d-b68257dd562e-system-cni-dir\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162340 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/343419e3-9da3-4649-8286-a105c42fc915-env-overrides\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-var-lib-kubelet\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162452 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef4c5948-06e9-4626-8545-e242dde01132-hosts-file\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162550 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-openvswitch\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162556 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/343419e3-9da3-4649-8286-a105c42fc915-run-ovn\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-systemd\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1feee572-8d94-45fe-8be3-b7348a5bfa66-dbus\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162650 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-sys\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.162671 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-modprobe-d\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.163177 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-run\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.163177 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.162775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4892e0-666c-462a-a2b0-98fca0d564b4-lib-modules\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.163177 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.163050 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.163177 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.163119 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d03e58df-c1b1-4de8-916d-b68257dd562e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.164502 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.164473 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-tmp\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.164805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.164720 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb4892e0-666c-462a-a2b0-98fca0d564b4-etc-tuned\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.164985 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.164937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/343419e3-9da3-4649-8286-a105c42fc915-ovn-node-metrics-cert\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.168029 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.168006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bd4m\" (UniqueName: \"kubernetes.io/projected/fb4892e0-666c-462a-a2b0-98fca0d564b4-kube-api-access-5bd4m\") pod \"tuned-dlbwd\" (UID: \"fb4892e0-666c-462a-a2b0-98fca0d564b4\") " pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.172605 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.172579 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:58:54.172726 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.172612 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:58:54.172726 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.172627 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:54.172726 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.172705 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:54.672668513 +0000 UTC m=+3.047331684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:54.172726 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.172719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wkg7\" (UniqueName: \"kubernetes.io/projected/d03e58df-c1b1-4de8-916d-b68257dd562e-kube-api-access-2wkg7\") pod \"multus-additional-cni-plugins-9gglm\" (UID: \"d03e58df-c1b1-4de8-916d-b68257dd562e\") " pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.175826 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.175793 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wfr\" (UniqueName: \"kubernetes.io/projected/ef4c5948-06e9-4626-8545-e242dde01132-kube-api-access-q7wfr\") pod \"node-resolver-bvbtd\" (UID: \"ef4c5948-06e9-4626-8545-e242dde01132\") " pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.176014 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.175985 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kplm\" (UniqueName: \"kubernetes.io/projected/343419e3-9da3-4649-8286-a105c42fc915-kube-api-access-8kplm\") pod \"ovnkube-node-g55gm\" (UID: \"343419e3-9da3-4649-8286-a105c42fc915\") " pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.176430 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.176378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7xn\" (UniqueName: \"kubernetes.io/projected/84d1c8dc-cebf-4e8e-9f51-0f97ec557806-kube-api-access-8n7xn\") pod \"iptables-alerter-dm92m\" (UID: \"84d1c8dc-cebf-4e8e-9f51-0f97ec557806\") " pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.260396 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-k8s-cni-cncf-io\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bnh\" (UniqueName: \"kubernetes.io/projected/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-kube-api-access-98bnh\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-registration-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260459 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-k8s-cni-cncf-io\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae691532-a868-4769-bde5-c93f7eb4e1d0-host\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae691532-a868-4769-bde5-c93f7eb4e1d0-serviceca\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.260558 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260537 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-registration-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-system-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-bin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-kubelet\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-hostroot\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-system-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260657 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae691532-a868-4769-bde5-c93f7eb4e1d0-host\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-netns\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260713 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-netns\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260726 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-bin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-conf-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-kubelet\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260767 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-hostroot\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-daemon-config\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.260805 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260791 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-conf-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260816 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qkb\" (UniqueName: \"kubernetes.io/projected/ae691532-a868-4769-bde5-c93f7eb4e1d0-kube-api-access-69qkb\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-etc-kubernetes\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2rl\" (UniqueName: \"kubernetes.io/projected/8436bc20-fc7b-4845-b5df-7b681ee5fee0-kube-api-access-9j2rl\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260949 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-socket-dir-parent\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-etc-kubernetes\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.260992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9447ea52-6b76-4675-a2fd-447b95a0b740-agent-certs\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-multus\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-sys-fs\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261092 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpm6\" (UniqueName: \"kubernetes.io/projected/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kube-api-access-2cpm6\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae691532-a868-4769-bde5-c93f7eb4e1d0-serviceca\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-device-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261196 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9447ea52-6b76-4675-a2fd-447b95a0b740-konnectivity-ca\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261172 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-var-lib-cni-multus\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-socket-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cnibin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-sys-fs\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cni-binary-copy\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-multus-certs\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261303 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-os-release\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261368 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-daemon-config\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-os-release\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-socket-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261610 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5eb2019-f88d-456e-bba8-7ed2a59dc842-device-dir\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-host-run-multus-certs\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.261737 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261706 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-socket-dir-parent\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.261779 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-multus-cni-dir\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.261844 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:58:54.761827108 +0000 UTC m=+3.136490266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261857 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cnibin\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.261917 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8436bc20-fc7b-4845-b5df-7b681ee5fee0-cni-binary-copy\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.262175 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.262102 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9447ea52-6b76-4675-a2fd-447b95a0b740-konnectivity-ca\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.263289 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.263264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9447ea52-6b76-4675-a2fd-447b95a0b740-agent-certs\") pod \"konnectivity-agent-jswkv\" (UID: \"9447ea52-6b76-4675-a2fd-447b95a0b740\") " pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.272132 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.272106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpm6\" (UniqueName: \"kubernetes.io/projected/f5eb2019-f88d-456e-bba8-7ed2a59dc842-kube-api-access-2cpm6\") pod \"aws-ebs-csi-driver-node-xtrl7\" (UID: \"f5eb2019-f88d-456e-bba8-7ed2a59dc842\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.272783 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.272758 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2rl\" (UniqueName: \"kubernetes.io/projected/8436bc20-fc7b-4845-b5df-7b681ee5fee0-kube-api-access-9j2rl\") pod \"multus-8bbvd\" (UID: \"8436bc20-fc7b-4845-b5df-7b681ee5fee0\") " pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.272923 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.272904 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bnh\" (UniqueName: \"kubernetes.io/projected/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-kube-api-access-98bnh\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.272923 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.272915 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qkb\" (UniqueName: \"kubernetes.io/projected/ae691532-a868-4769-bde5-c93f7eb4e1d0-kube-api-access-69qkb\") pod \"node-ca-fjkxn\" (UID: \"ae691532-a868-4769-bde5-c93f7eb4e1d0\") " pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.338955 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.338867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bvbtd" Jul 23 17:58:54.347951 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.347928 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gglm" Jul 23 17:58:54.360774 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.360752 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm92m" Jul 23 17:58:54.369301 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.369276 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:58:54.375972 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.375953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" Jul 23 17:58:54.384512 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.384494 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" Jul 23 17:58:54.392128 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.392106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjkxn" Jul 23 17:58:54.399745 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.399725 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:58:54.407514 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.407494 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8bbvd" Jul 23 17:58:54.663363 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.663332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:54.663526 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.663450 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:54.663526 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.663503 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:55.663490664 +0000 UTC m=+4.038153822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:54.764408 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.764373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:54.764582 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:54.764561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:54.764666 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764564 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:54.764730 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764659 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:58:54.764730 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764706 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:58:54.764730 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764723 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:54.764856 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764671 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:58:55.764654919 +0000 UTC m=+4.139318092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:54.764856 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:54.764804 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:55.764787274 +0000 UTC m=+4.139450434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:55.114906 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:55.114798 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:53:54 +0000 UTC" deadline="2028-04-12 08:50:59.925859198 +0000 UTC" Jul 23 17:58:55.114906 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:55.114856 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15086h52m4.811008156s" Jul 23 17:58:55.262438 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.262391 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d1c8dc_cebf_4e8e_9f51_0f97ec557806.slice/crio-161f7f32b7fe333b3feab0301663a0ac7f8c27fe72a338674065e0320dacf3b5 WatchSource:0}: Error finding container 161f7f32b7fe333b3feab0301663a0ac7f8c27fe72a338674065e0320dacf3b5: Status 404 returned error can't find the container with id 161f7f32b7fe333b3feab0301663a0ac7f8c27fe72a338674065e0320dacf3b5 Jul 23 17:58:55.264151 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.264111 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4892e0_666c_462a_a2b0_98fca0d564b4.slice/crio-d365350306af12465c399701ee58a5c9861053103e5c6ad896a8b0b95b589b8e WatchSource:0}: Error finding container d365350306af12465c399701ee58a5c9861053103e5c6ad896a8b0b95b589b8e: Status 404 returned error can't find the container with id d365350306af12465c399701ee58a5c9861053103e5c6ad896a8b0b95b589b8e Jul 23 17:58:55.265257 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.265238 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae691532_a868_4769_bde5_c93f7eb4e1d0.slice/crio-7eb1c0ece0d788a34b9cb7c335cb3fad5fc6276ef7b8eb844d01d39103fa7ef8 WatchSource:0}: Error finding container 7eb1c0ece0d788a34b9cb7c335cb3fad5fc6276ef7b8eb844d01d39103fa7ef8: Status 404 returned error can't find the container with id 7eb1c0ece0d788a34b9cb7c335cb3fad5fc6276ef7b8eb844d01d39103fa7ef8 Jul 23 17:58:55.266446 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.266423 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4c5948_06e9_4626_8545_e242dde01132.slice/crio-1823d10d4abfd3eaf23d68bbe304425442fa1a5789e1f05dd737c30c410c7f49 WatchSource:0}: Error finding container 1823d10d4abfd3eaf23d68bbe304425442fa1a5789e1f05dd737c30c410c7f49: Status 404 returned error can't find the container with id 1823d10d4abfd3eaf23d68bbe304425442fa1a5789e1f05dd737c30c410c7f49 Jul 23 17:58:55.268005 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.267981 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8436bc20_fc7b_4845_b5df_7b681ee5fee0.slice/crio-90c8dc6f95e3c403de697760d5ada4e67b0b3962912aef87747dd99798a3a3cf WatchSource:0}: Error finding container 90c8dc6f95e3c403de697760d5ada4e67b0b3962912aef87747dd99798a3a3cf: Status 404 returned error can't find the container with id 90c8dc6f95e3c403de697760d5ada4e67b0b3962912aef87747dd99798a3a3cf Jul 23 17:58:55.269327 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.269171 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5eb2019_f88d_456e_bba8_7ed2a59dc842.slice/crio-1e36a10a8e53f853c2f4bada18d1f745bc018eeb6fd8265ccf72917f165ac5b1 WatchSource:0}: Error finding container 1e36a10a8e53f853c2f4bada18d1f745bc018eeb6fd8265ccf72917f165ac5b1: Status 404 returned error can't find the container with id 1e36a10a8e53f853c2f4bada18d1f745bc018eeb6fd8265ccf72917f165ac5b1 Jul 23 17:58:55.273222 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.273195 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9447ea52_6b76_4675_a2fd_447b95a0b740.slice/crio-5d824d7aaf1c9da364358350380612120d5f14ce4e7dfa357c3b91bf7c8073e7 WatchSource:0}: Error finding container 5d824d7aaf1c9da364358350380612120d5f14ce4e7dfa357c3b91bf7c8073e7: Status 404 returned error can't find the container with id 5d824d7aaf1c9da364358350380612120d5f14ce4e7dfa357c3b91bf7c8073e7 Jul 23 17:58:55.273989 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.273955 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03e58df_c1b1_4de8_916d_b68257dd562e.slice/crio-b9565966a75280f13a4e0aab21370893f45b1f8edc76ccbe67da9e3a6affe36a WatchSource:0}: Error finding container b9565966a75280f13a4e0aab21370893f45b1f8edc76ccbe67da9e3a6affe36a: Status 404 returned error can't find the container with id b9565966a75280f13a4e0aab21370893f45b1f8edc76ccbe67da9e3a6affe36a Jul 23 17:58:55.275944 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:58:55.275577 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343419e3_9da3_4649_8286_a105c42fc915.slice/crio-14773d50d710351b25702d147a4513978097045071da9f202d963d5a75f91009 WatchSource:0}: Error finding container 14773d50d710351b25702d147a4513978097045071da9f202d963d5a75f91009: Status 404 returned error can't find the container with id 14773d50d710351b25702d147a4513978097045071da9f202d963d5a75f91009 Jul 23 17:58:55.669862 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:55.669806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:55.670031 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.669973 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:55.670086 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.670046 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:57.670027915 +0000 UTC m=+6.044691095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:55.770958 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:55.770868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:55.770958 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:55.770955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:55.771163 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771098 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:55.771163 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771160 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:58:57.771141888 +0000 UTC m=+6.145805057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:55.771578 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771558 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:58:55.771671 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771587 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:58:55.771671 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771599 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:55.771671 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:55.771647 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:57.771631024 +0000 UTC m=+6.146294188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:56.115699 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.115586 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:53:54 +0000 UTC" deadline="2028-01-17 17:52:05.4638271 +0000 UTC" Jul 23 17:58:56.115699 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.115621 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13031h53m9.348210694s" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.151806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.151846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:56.151964 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.152129 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:56.152265 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:58:56.152409 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:56.152402 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:58:56.166411 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.166373 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8bbvd" event={"ID":"8436bc20-fc7b-4845-b5df-7b681ee5fee0","Type":"ContainerStarted","Data":"90c8dc6f95e3c403de697760d5ada4e67b0b3962912aef87747dd99798a3a3cf"} Jul 23 17:58:56.171492 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.171458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bvbtd" event={"ID":"ef4c5948-06e9-4626-8545-e242dde01132","Type":"ContainerStarted","Data":"1823d10d4abfd3eaf23d68bbe304425442fa1a5789e1f05dd737c30c410c7f49"} Jul 23 17:58:56.181456 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.181326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" event={"ID":"fb4892e0-666c-462a-a2b0-98fca0d564b4","Type":"ContainerStarted","Data":"d365350306af12465c399701ee58a5c9861053103e5c6ad896a8b0b95b589b8e"} Jul 23 17:58:56.186091 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.186037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjkxn" event={"ID":"ae691532-a868-4769-bde5-c93f7eb4e1d0","Type":"ContainerStarted","Data":"7eb1c0ece0d788a34b9cb7c335cb3fad5fc6276ef7b8eb844d01d39103fa7ef8"} Jul 23 17:58:56.201228 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.201165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerStarted","Data":"b9565966a75280f13a4e0aab21370893f45b1f8edc76ccbe67da9e3a6affe36a"} Jul 23 17:58:56.204408 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.204334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jswkv" event={"ID":"9447ea52-6b76-4675-a2fd-447b95a0b740","Type":"ContainerStarted","Data":"5d824d7aaf1c9da364358350380612120d5f14ce4e7dfa357c3b91bf7c8073e7"} Jul 23 17:58:56.206598 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.206525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm92m" event={"ID":"84d1c8dc-cebf-4e8e-9f51-0f97ec557806","Type":"ContainerStarted","Data":"161f7f32b7fe333b3feab0301663a0ac7f8c27fe72a338674065e0320dacf3b5"} Jul 23 17:58:56.220716 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.220080 2568 generic.go:356] "Generic (PLEG): container finished" podID="4c2570112e3a53f32ff7db5d034b5370" containerID="9b531cc551dc783d66e1b6105ffa28b0f272d2ec8fa9fe931dff9f5564bdfaeb" exitCode=0 Jul 23 17:58:56.220716 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.220171 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"4c2570112e3a53f32ff7db5d034b5370","Type":"ContainerDied","Data":"9b531cc551dc783d66e1b6105ffa28b0f272d2ec8fa9fe931dff9f5564bdfaeb"} Jul 23 17:58:56.249235 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.248754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" event={"ID":"100d0bec6eaa8166829fa6f437da8a36","Type":"ContainerStarted","Data":"8bab99f6235a738b8e1bf63b77e060c510ea6e94c0ed3c93ba10570f3812fc87"} Jul 23 17:58:56.252019 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.251989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"14773d50d710351b25702d147a4513978097045071da9f202d963d5a75f91009"} Jul 23 17:58:56.264596 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:56.264563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" event={"ID":"f5eb2019-f88d-456e-bba8-7ed2a59dc842","Type":"ContainerStarted","Data":"1e36a10a8e53f853c2f4bada18d1f745bc018eeb6fd8265ccf72917f165ac5b1"} Jul 23 17:58:57.283011 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:57.282949 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" event={"ID":"4c2570112e3a53f32ff7db5d034b5370","Type":"ContainerStarted","Data":"98a53606d3d36608e4a7a905d35153efbb620612b0d043ca97e647e1ffeb291f"} Jul 23 17:58:57.299230 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:57.299125 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-67.ec2.internal" podStartSLOduration=4.299103731 podStartE2EDuration="4.299103731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:56.265262502 +0000 UTC m=+4.639925681" watchObservedRunningTime="2026-07-23 17:58:57.299103731 +0000 UTC m=+5.673766929" Jul 23 17:58:57.687017 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:57.686339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:57.687017 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.686555 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:57.687017 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.686619 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:01.686599925 +0000 UTC m=+10.061263105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:57.787353 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:57.787312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:57.787605 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:57.787386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:57.787605 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.787535 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:57.787605 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.787606 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:59:01.787585167 +0000 UTC m=+10.162248338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:57.788067 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.788046 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:58:57.788160 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.788074 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:58:57.788160 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.788089 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:57.788160 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:57.788137 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:01.788120317 +0000 UTC m=+10.162783474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:58.149672 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:58.149632 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:58:58.149882 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:58.149791 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:58:58.150225 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:58.150204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:58:58.150320 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:58.150306 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:58:58.150383 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:58:58.150359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:58:58.150469 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:58:58.150438 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:00.153796 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:00.153764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:00.154263 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:00.153939 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:00.154464 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:00.154446 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:00.154630 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:00.154606 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:00.154796 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:00.154782 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:00.154937 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:00.154908 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:01.725122 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:01.725014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:01.725586 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.725140 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:01.725586 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.725206 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:09.725187456 +0000 UTC m=+18.099850613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:01.825938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:01.826022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826173 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826194 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826207 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826263 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:09.826245933 +0000 UTC m=+18.200909113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:01.826595 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826511 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:01.827093 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:01.826627 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:59:09.826553054 +0000 UTC m=+18.201216226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:02.147980 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:02.147947 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:02.148116 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:02.148075 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:02.148406 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:02.148389 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:02.148492 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:02.148473 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:02.148599 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:02.148586 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:02.148675 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:02.148652 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:04.147416 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:04.147379 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:04.147895 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:04.147379 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:04.147895 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:04.147517 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:04.147895 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:04.147634 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:04.147895 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:04.147660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:04.147895 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:04.147782 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:06.147316 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:06.147278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:06.147316 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:06.147305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:06.147807 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:06.147278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:06.147807 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:06.147411 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:06.147807 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:06.147459 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:06.147807 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:06.147520 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:08.147664 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:08.147628 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:08.148110 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:08.147628 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:08.148110 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:08.147781 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:08.148110 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:08.147628 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:08.148110 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:08.147812 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:08.148110 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:08.147927 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:09.795144 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:09.795105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:09.795637 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.795238 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:09.795637 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.795314 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.795294003 +0000 UTC m=+34.169957173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:09.895744 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:09.895708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:09.895930 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:09.895781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:09.895930 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.895901 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:09.895930 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.895923 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:59:09.896073 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.895945 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:59:09.896073 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.895958 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:09.896073 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.895980 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.895957471 +0000 UTC m=+34.270620627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:09.896073 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:09.896007 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.895990011 +0000 UTC m=+34.270653180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:10.146800 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:10.146765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:10.146984 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:10.146772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:10.146984 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:10.146885 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:10.146984 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:10.146965 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:10.147135 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:10.147019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:10.147177 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:10.147123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:12.147549 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:12.147514 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:12.147982 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:12.147550 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:12.147982 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:12.147666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:12.147982 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:12.147710 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:12.147982 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:12.147786 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:12.147982 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:12.147833 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:14.146949 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:14.146908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:14.147442 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:14.146913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:14.147442 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:14.147016 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:14.147442 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:14.146909 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:14.147442 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:14.147093 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:14.147442 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:14.147212 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:15.324030 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.323599 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"ad0e172da8169b6e5d47c5761bbf5d09ad431aac2072a93c28a4a0a92c0e75bd"} Jul 23 17:59:15.324861 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.324129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"70cf979e51997a7ccecc295e24a4fad127befe37fa31f5db2c7dbbd98af7400b"} Jul 23 17:59:15.324861 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.324178 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"e8df6fb35eeb614db35ef91d427ec5555e1821d3c27373d92a41d195c7e557d4"} Jul 23 17:59:15.324861 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.324223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"4eb06fe5c67158be4de8d0cdb3a910f4f9dd96b7df161d1a5284de7f5d298e45"} Jul 23 17:59:15.324861 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.324268 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"e378d8c4be24d446566dcf5c4761f201414f934c43c239906f351a0f33fe65e0"} Jul 23 17:59:15.324861 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.324317 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"61c71b79e3fcc76acc4f02fa9da86c44f34b00f8be10bfbbe9698e509ff7ebfb"} Jul 23 17:59:15.325432 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.325375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" event={"ID":"f5eb2019-f88d-456e-bba8-7ed2a59dc842","Type":"ContainerStarted","Data":"cb34ce59d7c5aad37b6295be6f91f219a27ecf7396074acc4d76f98342de9aa9"} Jul 23 17:59:15.327093 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.327065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8bbvd" event={"ID":"8436bc20-fc7b-4845-b5df-7b681ee5fee0","Type":"ContainerStarted","Data":"aaa3ed1cb8ebc33f60465507f09cfb9a12fee70009cd933a134adf6a2d0c0b7e"} Jul 23 17:59:15.329083 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.329058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bvbtd" event={"ID":"ef4c5948-06e9-4626-8545-e242dde01132","Type":"ContainerStarted","Data":"d0aded37ed7e93e562affbb7187da001c2f25fd79d0b57c47385423c3f258836"} Jul 23 17:59:15.330654 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.330629 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" event={"ID":"fb4892e0-666c-462a-a2b0-98fca0d564b4","Type":"ContainerStarted","Data":"ec3786dbabccf108783ab512b425fbf3c91fd8dbf376e381aa921f6485483c97"} Jul 23 17:59:15.332221 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.332197 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjkxn" event={"ID":"ae691532-a868-4769-bde5-c93f7eb4e1d0","Type":"ContainerStarted","Data":"7c4d7171c290e0d6f752fb65192f4a83eca778c315adf208836848edbe740cf8"} Jul 23 17:59:15.333651 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.333628 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="74996328831ad084ac85913f91290ac9d20583218d1b6e6f497d7b7780b15be0" exitCode=0 Jul 23 17:59:15.333766 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.333714 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"74996328831ad084ac85913f91290ac9d20583218d1b6e6f497d7b7780b15be0"} Jul 23 17:59:15.335490 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.335465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jswkv" event={"ID":"9447ea52-6b76-4675-a2fd-447b95a0b740","Type":"ContainerStarted","Data":"4d7ded8666b6b740eae4b7708e4aa25f0f0c208ba52a2430cc767aac1b48b35c"} Jul 23 17:59:15.346745 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.346241 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8bbvd" podStartSLOduration=4.245815805 podStartE2EDuration="23.346223296s" totalImagesPullingTime="19.100407491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:15.346162661 +0000 UTC m=+23.720825841" watchObservedRunningTime="2026-07-23 17:59:15.346223296 +0000 UTC m=+23.720886476" Jul 23 17:59:15.346745 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.346706 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-67.ec2.internal" podStartSLOduration=22.34667076 podStartE2EDuration="22.34667076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:57.299921777 +0000 UTC m=+5.674584962" watchObservedRunningTime="2026-07-23 17:59:15.34667076 +0000 UTC m=+23.721333944" Jul 23 17:59:15.361513 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.361450 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jswkv" podStartSLOduration=4.335235118 podStartE2EDuration="23.361428487s" totalImagesPullingTime="19.026193369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:15.360575536 +0000 UTC m=+23.735238723" watchObservedRunningTime="2026-07-23 17:59:15.361428487 +0000 UTC m=+23.736091667" Jul 23 17:59:15.401250 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.401201 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlbwd" podStartSLOduration=4.368117111 podStartE2EDuration="23.401185921s" totalImagesPullingTime="19.03306881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:15.400896616 +0000 UTC m=+23.775559800" watchObservedRunningTime="2026-07-23 17:59:15.401185921 +0000 UTC m=+23.775849100" Jul 23 17:59:15.417615 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.417569 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bvbtd" podStartSLOduration=4.379165517 podStartE2EDuration="23.417553269s" totalImagesPullingTime="19.038387752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:15.417097507 +0000 UTC m=+23.791760686" watchObservedRunningTime="2026-07-23 17:59:15.417553269 +0000 UTC m=+23.792216448" Jul 23 17:59:15.881097 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:15.880993 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 17:59:16.137059 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.136930 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T17:59:15.881089552Z","UUID":"fc55f62d-194b-4ce6-a080-240756fb9410","Handler":null,"Name":"","Endpoint":""} Jul 23 17:59:16.138792 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.138770 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 17:59:16.138931 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.138813 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 17:59:16.147613 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.147579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:16.147797 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.147585 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:16.147797 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:16.147767 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:16.147969 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.147944 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:16.148050 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:16.147961 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:16.148113 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:16.148059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:16.340991 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.340940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" event={"ID":"f5eb2019-f88d-456e-bba8-7ed2a59dc842","Type":"ContainerStarted","Data":"0620d4095feb9731f0df0ec9cb2a994ad60b0499e32552ed2d41aed87433b235"} Jul 23 17:59:16.343478 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.343375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm92m" event={"ID":"84d1c8dc-cebf-4e8e-9f51-0f97ec557806","Type":"ContainerStarted","Data":"97447e3998dcf248c85f9658d7d0a51c323341a2b0fcae8c7f77e25ed5eedaa4"} Jul 23 17:59:16.359210 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.359152 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fjkxn" podStartSLOduration=5.325683013 podStartE2EDuration="24.359137846s" totalImagesPullingTime="19.033454833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:15.444155849 +0000 UTC m=+23.818819027" watchObservedRunningTime="2026-07-23 17:59:16.359137846 +0000 UTC m=+24.733801058" Jul 23 17:59:16.359420 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:16.359332 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dm92m" podStartSLOduration=5.323296561 podStartE2EDuration="24.359325099s" totalImagesPullingTime="19.036028538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:16.358740927 +0000 UTC m=+24.733404109" watchObservedRunningTime="2026-07-23 17:59:16.359325099 +0000 UTC m=+24.733988279" Jul 23 17:59:17.349115 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:17.348771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"e1788a0ba86db42f30f7f5eca479907b70b9210e8d059a4423bed9e83bf86ee2"} Jul 23 17:59:17.351310 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:17.351257 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" event={"ID":"f5eb2019-f88d-456e-bba8-7ed2a59dc842","Type":"ContainerStarted","Data":"d385d97add01b40470b5f8c696adb0ce38536ff990d51341462ab84ac19757c7"} Jul 23 17:59:17.369669 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:17.369619 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtrl7" podStartSLOduration=4.118686752 podStartE2EDuration="25.369598669s" totalImagesPullingTime="21.250911917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:17.36921301 +0000 UTC m=+25.743876189" watchObservedRunningTime="2026-07-23 17:59:17.369598669 +0000 UTC m=+25.744261847" Jul 23 17:59:18.147107 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:18.147070 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:18.147107 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:18.147106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:18.147363 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:18.147071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:18.147363 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:18.147190 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:18.147363 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:18.147284 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:18.147503 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:18.147366 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:19.246891 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:19.246849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:59:19.248064 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:19.248044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:59:20.147897 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.147705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:20.148054 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.147707 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:20.148054 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:20.148007 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:20.148167 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.147707 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:20.148167 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:20.148073 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:20.148167 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:20.148156 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:20.361438 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.361407 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" event={"ID":"343419e3-9da3-4649-8286-a105c42fc915","Type":"ContainerStarted","Data":"0b61dee05b651b535cde59a410d4afc4a9e4b1f53c25e3914dbe7de4cfd4445a"} Jul 23 17:59:20.362155 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.361633 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:20.363199 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.363180 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="1dfa6ec0c03f1673395cc02676b3ec5e07c13ab614cdc12394dbbb9bc68dbc37" exitCode=0 Jul 23 17:59:20.363307 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.363212 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"1dfa6ec0c03f1673395cc02676b3ec5e07c13ab614cdc12394dbbb9bc68dbc37"} Jul 23 17:59:20.377496 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.377474 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:20.394855 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.394817 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" podStartSLOduration=9.305520871 podStartE2EDuration="28.39480255s" totalImagesPullingTime="19.089281679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:20.393057452 +0000 UTC m=+28.767720633" watchObservedRunningTime="2026-07-23 17:59:20.39480255 +0000 UTC m=+28.769465729" Jul 23 17:59:20.688542 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:20.688502 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:21.263561 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.263521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:59:21.263827 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.263717 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 17:59:21.264184 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.264166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jswkv" Jul 23 17:59:21.366870 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.366842 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:21.384904 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.384882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:21.670012 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.669980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv8b5"] Jul 23 17:59:21.670254 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.670114 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:21.670254 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:21.670232 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:21.672881 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.672853 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vlpnb"] Jul 23 17:59:21.672987 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.672972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:21.673092 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:21.673073 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:21.675633 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.675606 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s789k"] Jul 23 17:59:21.675749 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:21.675710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:21.675805 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:21.675781 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:22.370595 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:22.370558 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="52dd3e73954ea72ccb55b758a71764d2d0e11a803a0e019999430a66e30a8944" exitCode=0 Jul 23 17:59:22.370982 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:22.370642 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"52dd3e73954ea72ccb55b758a71764d2d0e11a803a0e019999430a66e30a8944"} Jul 23 17:59:23.146928 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:23.146891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:23.147128 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:23.146891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:23.147128 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:23.147004 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:23.147128 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:23.146893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:23.147128 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:23.147093 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:23.147314 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:23.147226 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:24.377031 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:24.376836 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="38898c2bbbe1810972494e1ec4e58ea3a8a371de6930ff9d182649a2bba5b93d" exitCode=0 Jul 23 17:59:24.377031 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:24.376927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"38898c2bbbe1810972494e1ec4e58ea3a8a371de6930ff9d182649a2bba5b93d"} Jul 23 17:59:25.146986 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.146952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:25.147155 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.146952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:25.147155 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.147097 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlpnb" podUID="5bdc93a0-ac64-4d07-ab67-96e3e6339cff" Jul 23 17:59:25.147155 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.146952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:25.147155 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.147134 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s789k" podUID="656afe83-c2a9-452f-ae53-3ea5138f28a2" Jul 23 17:59:25.147386 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.147234 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv8b5" podUID="1feee572-8d94-45fe-8be3-b7348a5bfa66" Jul 23 17:59:25.814377 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.814341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:25.814997 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.814478 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:25.814997 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.814544 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret podName:1feee572-8d94-45fe-8be3-b7348a5bfa66 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:57.81452753 +0000 UTC m=+66.189190691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret") pod "global-pull-secret-syncer-qv8b5" (UID: "1feee572-8d94-45fe-8be3-b7348a5bfa66") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:59:25.915381 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.915351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:25.915573 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:25.915400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:25.915573 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915564 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:25.915677 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915581 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:59:25.915677 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915600 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:59:25.915677 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915613 2568 projected.go:196] Error preparing data for projected volume kube-api-access-nh98b for pod openshift-network-diagnostics/network-check-target-s789k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:25.915677 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915632 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 17:59:57.915612764 +0000 UTC m=+66.290275941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:59:25.915897 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:25.915729 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b podName:656afe83-c2a9-452f-ae53-3ea5138f28a2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:57.915716756 +0000 UTC m=+66.290379917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh98b" (UniqueName: "kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b") pod "network-check-target-s789k" (UID: "656afe83-c2a9-452f-ae53-3ea5138f28a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:59:26.925863 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:26.925834 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-67.ec2.internal" event="NodeReady" Jul 23 17:59:26.926299 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:26.925957 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 17:59:26.971418 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:26.971386 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659fb474bb-62jbx"] Jul 23 17:59:26.976286 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:26.976263 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:26.978934 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.978884 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" Jul 23 17:59:26.979390 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.979315 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-4ddxk\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4ddxk\"" type="*v1.Secret" Jul 23 17:59:26.979390 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.979330 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"image-registry-659fb474bb-62jbx\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" podUID="cfccd801-4b47-4f7e-bbac-92e67e3e18de" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:26.979390 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.979359 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" Jul 23 17:59:26.979556 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.979390 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" Jul 23 17:59:26.979556 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:26.979409 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-129-67.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-129-67.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" Jul 23 17:59:26.995796 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:26.995730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659fb474bb-62jbx"] Jul 23 17:59:27.001677 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.001651 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qnj5r"] Jul 23 17:59:27.003514 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.003484 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.006357 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.006310 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 17:59:27.006357 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.006333 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 17:59:27.007116 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.007100 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k2gdw\"" Jul 23 17:59:27.007225 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.007102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 17:59:27.019979 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.019949 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnj5r"] Jul 23 17:59:27.084017 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.083936 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j7q2q"] Jul 23 17:59:27.086049 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.086022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.088500 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.088476 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 17:59:27.088623 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.088585 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 17:59:27.088623 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.088599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb24m\"" Jul 23 17:59:27.096223 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.096201 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j7q2q"] Jul 23 17:59:27.125042 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmc52\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125042 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125271 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125271 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125271 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125393 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125393 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.125393 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpks\" (UniqueName: \"kubernetes.io/projected/0437e524-41b9-4b65-a9aa-6f24c560e36d-kube-api-access-9gpks\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.125505 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0437e524-41b9-4b65-a9aa-6f24c560e36d-cert\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.125505 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.125458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.147367 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.147335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:27.147367 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.147366 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:27.147564 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.147369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:27.150119 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.149920 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:59:27.150119 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.149939 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:59:27.150119 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.149964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:59:27.150119 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.150003 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:59:27.150119 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.149942 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbh2j\"" Jul 23 17:59:27.150383 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.150333 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hv4kc\"" Jul 23 17:59:27.225862 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.225818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmc52\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.225881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.225914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.225954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-config-volume\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.225980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226056 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-tmp-dir\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226187 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpks\" (UniqueName: \"kubernetes.io/projected/0437e524-41b9-4b65-a9aa-6f24c560e36d-kube-api-access-9gpks\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p57\" (UniqueName: \"kubernetes.io/projected/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-kube-api-access-j5p57\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.226287 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0437e524-41b9-4b65-a9aa-6f24c560e36d-cert\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.226502 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226730 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226707 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.226730 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.226721 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.231232 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.231194 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0437e524-41b9-4b65-a9aa-6f24c560e36d-cert\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.237356 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.237323 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.237460 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.237399 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmc52\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:27.237553 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.237533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpks\" (UniqueName: \"kubernetes.io/projected/0437e524-41b9-4b65-a9aa-6f24c560e36d-kube-api-access-9gpks\") pod \"ingress-canary-qnj5r\" (UID: \"0437e524-41b9-4b65-a9aa-6f24c560e36d\") " pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.314425 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.314387 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnj5r" Jul 23 17:59:27.327328 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-config-volume\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.327482 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.327482 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:27.327457 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:27.327597 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-tmp-dir\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.327597 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:27.327530 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:27.827509655 +0000 UTC m=+36.202172826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:27.327597 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p57\" (UniqueName: \"kubernetes.io/projected/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-kube-api-access-j5p57\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.327812 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327780 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-tmp-dir\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.328028 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.327995 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-config-volume\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.338577 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.338492 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p57\" (UniqueName: \"kubernetes.io/projected/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-kube-api-access-j5p57\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.832467 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.832421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:27.832630 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:27.832595 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:27.832713 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:27.832702 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:28.83266653 +0000 UTC m=+37.207329688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:27.846820 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:27.846787 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4ddxk\"" Jul 23 17:59:28.079960 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.079931 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 17:59:28.087229 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.087151 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:28.087229 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.087181 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:28.087442 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.087255 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:28.587234584 +0000 UTC m=+36.961897743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:28.162265 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.162236 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 17:59:28.168921 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.168892 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.226662 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.226454 2568 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: failed to sync secret cache: timed out waiting for the condition Jul 23 17:59:28.226874 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.226463 2568 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: failed to sync configmap cache: timed out waiting for the condition Jul 23 17:59:28.226874 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.226739 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:28.726718356 +0000 UTC m=+37.101381518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : failed to sync secret cache: timed out waiting for the condition Jul 23 17:59:28.226874 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.226840 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:28.726822475 +0000 UTC m=+37.101485632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : failed to sync configmap cache: timed out waiting for the condition Jul 23 17:59:28.252319 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.252285 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 17:59:28.379005 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.378983 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 17:59:28.484053 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.484025 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnj5r"] Jul 23 17:59:28.488366 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:59:28.488336 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0437e524_41b9_4b65_a9aa_6f24c560e36d.slice/crio-98dbf3d99763cda3c289263d98c0b7e1ab064db703f1d7149bfd5bbd9e70bd8f WatchSource:0}: Error finding container 98dbf3d99763cda3c289263d98c0b7e1ab064db703f1d7149bfd5bbd9e70bd8f: Status 404 returned error can't find the container with id 98dbf3d99763cda3c289263d98c0b7e1ab064db703f1d7149bfd5bbd9e70bd8f Jul 23 17:59:28.638400 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.638369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.638570 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.638547 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:28.638632 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.638576 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:28.638677 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.638646 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:29.638624682 +0000 UTC m=+38.013287857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:28.739773 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.739723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.739773 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.739786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.740620 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.740597 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.743329 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.743305 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:28.840583 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:28.840544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:28.840760 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.840720 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:28.840799 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:28.840786 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:30.840771684 +0000 UTC m=+39.215434841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:29.390128 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:29.390088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnj5r" event={"ID":"0437e524-41b9-4b65-a9aa-6f24c560e36d","Type":"ContainerStarted","Data":"98dbf3d99763cda3c289263d98c0b7e1ab064db703f1d7149bfd5bbd9e70bd8f"} Jul 23 17:59:29.393231 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:29.393202 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="d065744b91d3c9f2ec974f7506278b88c7a8889017c0671d84760480b9d3ed53" exitCode=0 Jul 23 17:59:29.393334 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:29.393241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"d065744b91d3c9f2ec974f7506278b88c7a8889017c0671d84760480b9d3ed53"} Jul 23 17:59:29.646918 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:29.646657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:29.647058 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:29.646822 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:29.647058 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:29.646956 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:29.647058 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:29.647028 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:31.647007029 +0000 UTC m=+40.021670206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:30.134965 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:30.134928 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:30.275272 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:30.275162 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:30.399742 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:30.399654 2568 generic.go:356] "Generic (PLEG): container finished" podID="d03e58df-c1b1-4de8-916d-b68257dd562e" containerID="e4dfa1d363079e899ec018725590e066305c1f4ee583a5c0ed588d02c67798bc" exitCode=0 Jul 23 17:59:30.399742 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:30.399719 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerDied","Data":"e4dfa1d363079e899ec018725590e066305c1f4ee583a5c0ed588d02c67798bc"} Jul 23 17:59:30.856488 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:30.856459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:30.856601 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:30.856563 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:30.856648 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:30.856609 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:34.856595313 +0000 UTC m=+43.231258469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:31.403180 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.403145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnj5r" event={"ID":"0437e524-41b9-4b65-a9aa-6f24c560e36d","Type":"ContainerStarted","Data":"7d66c88264e5d5bafd3acadd4b4e2e27d3fc1e5cd9eb9c69adabbf64cad65b73"} Jul 23 17:59:31.406304 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.406275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gglm" event={"ID":"d03e58df-c1b1-4de8-916d-b68257dd562e","Type":"ContainerStarted","Data":"4e9d652ebe945d7de99cabe55643e2c7e74fea49fea7dce4b5e84ea89056b3ff"} Jul 23 17:59:31.420812 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.420765 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qnj5r" podStartSLOduration=3.045505647 podStartE2EDuration="5.420750611s" totalImagesPullingTime="2.375244964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:31.420049325 +0000 UTC m=+39.794712517" watchObservedRunningTime="2026-07-23 17:59:31.420750611 +0000 UTC m=+39.795413780" Jul 23 17:59:31.443496 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.443447 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9gglm" podStartSLOduration=10.118784575 podStartE2EDuration="39.443434434s" totalImagesPullingTime="29.324649859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:31.442097864 +0000 UTC m=+39.816761044" watchObservedRunningTime="2026-07-23 17:59:31.443434434 +0000 UTC m=+39.818097636" Jul 23 17:59:31.566238 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.566188 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:31.663751 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:31.663657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:31.663900 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:31.663869 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:31.663900 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:31.663897 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:31.663989 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:31.663976 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:35.663953348 +0000 UTC m=+44.038616529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:32.651294 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:32.651255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:34.884782 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:34.884738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:34.885175 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:34.884903 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:34.885175 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:34.884974 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:42.884955162 +0000 UTC m=+51.259618319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:35.690571 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:35.690526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:35.690797 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:35.690709 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:35.690797 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:35.690733 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:35.690797 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:35.690792 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:43.690775404 +0000 UTC m=+52.065438561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:39.243871 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:39.243834 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:39.387395 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:39.387343 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:39.609220 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:39.609114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:39.854989 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:39.854945 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:40.099297 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:40.099265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:40.246293 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:40.246258 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:40.342479 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:40.342442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:40.592643 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:40.592607 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:40.697619 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:40.697580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" limit=1024 Jul 23 17:59:42.938394 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:42.938358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:42.938831 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:42.938478 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:42.938831 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:42.938539 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 17:59:58.938523468 +0000 UTC m=+67.313186628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:43.742818 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:43.742775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:43.742999 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:43.742930 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:43.742999 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:43.742951 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:43.743076 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:43.743007 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 17:59:59.742990586 +0000 UTC m=+68.117653746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 17:59:53.385844 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:53.385817 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g55gm" Jul 23 17:59:57.841233 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.841181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:57.843851 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.843830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:59:57.857882 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.855127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1feee572-8d94-45fe-8be3-b7348a5bfa66-original-pull-secret\") pod \"global-pull-secret-syncer-qv8b5\" (UID: \"1feee572-8d94-45fe-8be3-b7348a5bfa66\") " pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:57.942271 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.942216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:57.942458 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.942315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 17:59:57.944965 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.944946 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:59:57.945041 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.945004 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:59:57.953040 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:57.953014 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 17:59:57.953133 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:57.953091 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs podName:5bdc93a0-ac64-4d07-ab67-96e3e6339cff nodeName:}" failed. No retries permitted until 2026-07-23 18:01:01.953069938 +0000 UTC m=+130.327733108 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs") pod "network-metrics-daemon-vlpnb" (UID: "5bdc93a0-ac64-4d07-ab67-96e3e6339cff") : secret "metrics-daemon-secret" not found Jul 23 17:59:57.955345 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.955323 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:59:57.965538 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:57.965512 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh98b\" (UniqueName: \"kubernetes.io/projected/656afe83-c2a9-452f-ae53-3ea5138f28a2-kube-api-access-nh98b\") pod \"network-check-target-s789k\" (UID: \"656afe83-c2a9-452f-ae53-3ea5138f28a2\") " pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:58.066209 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.066162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv8b5" Jul 23 17:59:58.076829 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.076809 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbh2j\"" Jul 23 17:59:58.085071 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.085051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 17:59:58.214407 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.214379 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv8b5"] Jul 23 17:59:58.218429 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:59:58.218404 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1feee572_8d94_45fe_8be3_b7348a5bfa66.slice/crio-db4d2124fd33f063267cb449ed00cb2b40f05a12b2c66922f9a5a4e9663e3745 WatchSource:0}: Error finding container db4d2124fd33f063267cb449ed00cb2b40f05a12b2c66922f9a5a4e9663e3745: Status 404 returned error can't find the container with id db4d2124fd33f063267cb449ed00cb2b40f05a12b2c66922f9a5a4e9663e3745 Jul 23 17:59:58.226608 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.226582 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s789k"] Jul 23 17:59:58.229572 ip-10-0-129-67 kubenswrapper[2568]: W0723 17:59:58.229550 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656afe83_c2a9_452f_ae53_3ea5138f28a2.slice/crio-d491546ec8aa7e96d095647320096f202e7273619328f4f3c94b68b19d69de97 WatchSource:0}: Error finding container d491546ec8aa7e96d095647320096f202e7273619328f4f3c94b68b19d69de97: Status 404 returned error can't find the container with id d491546ec8aa7e96d095647320096f202e7273619328f4f3c94b68b19d69de97 Jul 23 17:59:58.473909 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.473874 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv8b5" event={"ID":"1feee572-8d94-45fe-8be3-b7348a5bfa66","Type":"ContainerStarted","Data":"db4d2124fd33f063267cb449ed00cb2b40f05a12b2c66922f9a5a4e9663e3745"} Jul 23 17:59:58.476960 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.476916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s789k" event={"ID":"656afe83-c2a9-452f-ae53-3ea5138f28a2","Type":"ContainerStarted","Data":"d491546ec8aa7e96d095647320096f202e7273619328f4f3c94b68b19d69de97"} Jul 23 17:59:58.949449 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:58.949415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 17:59:58.949913 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:58.949603 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:59:58.949913 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:58.949702 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 18:00:30.949660373 +0000 UTC m=+99.324323533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 17:59:59.755835 ip-10-0-129-67 kubenswrapper[2568]: I0723 17:59:59.755796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 17:59:59.756014 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:59.755951 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:59:59.756014 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:59.755969 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 17:59:59.756129 ip-10-0-129-67 kubenswrapper[2568]: E0723 17:59:59.756023 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 18:00:31.756007859 +0000 UTC m=+100.130671020 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 18:00:03.491425 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:03.491384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s789k" event={"ID":"656afe83-c2a9-452f-ae53-3ea5138f28a2","Type":"ContainerStarted","Data":"2ea63525ae55389136466148a9a49d649b3a8e8d2059d0009b167aee5eb58358"} Jul 23 18:00:03.491940 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:03.491487 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 18:00:03.492828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:03.492806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv8b5" event={"ID":"1feee572-8d94-45fe-8be3-b7348a5bfa66","Type":"ContainerStarted","Data":"529e735aa4c05399be21ae3fb97610a63480a96464581e1b2a37e5a32d365f32"} Jul 23 18:00:03.510886 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:03.510840 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s789k" podStartSLOduration=67.112781323 podStartE2EDuration="1m11.510825718s" totalImagesPullingTime="4.398044395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:03.509400439 +0000 UTC m=+71.884063617" watchObservedRunningTime="2026-07-23 18:00:03.510825718 +0000 UTC m=+71.885488876" Jul 23 18:00:03.526395 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:03.526351 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qv8b5" podStartSLOduration=67.113990022 podStartE2EDuration="1m11.526337162s" totalImagesPullingTime="4.41234714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:03.525743264 +0000 UTC m=+71.900406443" watchObservedRunningTime="2026-07-23 18:00:03.526337162 +0000 UTC m=+71.901000341" Jul 23 18:00:11.609720 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.609666 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg"] Jul 23 18:00:11.611894 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.611877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.614520 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.614499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 18:00:11.614901 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.614884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pc7km\"" Jul 23 18:00:11.615523 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.615505 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:00:11.615607 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.615579 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 18:00:11.620803 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.620782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg"] Jul 23 18:00:11.643295 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.643264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.643295 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.643297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfshq\" (UniqueName: \"kubernetes.io/projected/faa76306-034d-4627-ae32-0ca13712c750-kube-api-access-sfshq\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.699283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.699256 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k"] Jul 23 18:00:11.701277 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.701255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.710567 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.710548 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 18:00:11.711035 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.711020 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 18:00:11.711124 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.711108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-62lqp\"" Jul 23 18:00:11.711871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.711856 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 18:00:11.712399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.712386 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 18:00:11.718322 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.718303 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k"] Jul 23 18:00:11.743673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.743639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.743673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.743674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8r9d\" (UniqueName: \"kubernetes.io/projected/febde259-4b55-421f-a4ec-1b4b78af5eac-kube-api-access-n8r9d\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.743921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.743799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.743921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.743832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfshq\" (UniqueName: \"kubernetes.io/projected/faa76306-034d-4627-ae32-0ca13712c750-kube-api-access-sfshq\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.743921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.743869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/febde259-4b55-421f-a4ec-1b4b78af5eac-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.744061 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:11.743923 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:11.744061 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:11.743978 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:12.243963331 +0000 UTC m=+80.618626487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:11.754622 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.754599 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfshq\" (UniqueName: \"kubernetes.io/projected/faa76306-034d-4627-ae32-0ca13712c750-kube-api-access-sfshq\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:11.844903 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.844863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/febde259-4b55-421f-a4ec-1b4b78af5eac-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.845107 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.844947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.845107 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.844972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8r9d\" (UniqueName: \"kubernetes.io/projected/febde259-4b55-421f-a4ec-1b4b78af5eac-kube-api-access-n8r9d\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.845107 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:11.845086 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:11.845252 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:11.845160 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:12.345144398 +0000 UTC m=+80.719807559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:11.845572 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.845550 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/febde259-4b55-421f-a4ec-1b4b78af5eac-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:11.856615 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:11.856585 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8r9d\" (UniqueName: \"kubernetes.io/projected/febde259-4b55-421f-a4ec-1b4b78af5eac-kube-api-access-n8r9d\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:12.247333 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:12.247295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:12.247498 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:12.247447 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:12.247537 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:12.247518 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:13.247500728 +0000 UTC m=+81.622163902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:12.347765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:12.347708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:12.347887 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:12.347861 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:12.347936 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:12.347926 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:13.347910061 +0000 UTC m=+81.722573243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:13.255854 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:13.255806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:13.256253 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:13.255981 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:13.256253 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:13.256058 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:15.256043036 +0000 UTC m=+83.630706194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:13.357122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:13.357067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:13.357264 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:13.357225 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:13.357303 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:13.357296 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:15.357279919 +0000 UTC m=+83.731943093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:15.273095 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:15.272946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:15.273569 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:15.273117 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:15.273569 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:15.273199 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:19.273176107 +0000 UTC m=+87.647839263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:15.374158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:15.374112 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:15.374306 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:15.374269 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:15.374357 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:15.374344 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:19.374328183 +0000 UTC m=+87.748991342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:16.546890 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.546855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hxkrb"] Jul 23 18:00:16.549415 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.549394 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.553885 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.553865 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mq246\"" Jul 23 18:00:16.555041 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.555015 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 18:00:16.555147 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.555121 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 18:00:16.555280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.555263 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 18:00:16.555340 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.555136 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 18:00:16.603802 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.603765 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxkrb"] Jul 23 18:00:16.684500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.684449 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdcx\" (UniqueName: \"kubernetes.io/projected/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-api-access-kjdcx\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.684743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.684521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f5fa97-3a75-482a-a435-5dff531f81e0-data-volume\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.684743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.684580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.684743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.684631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43f5fa97-3a75-482a-a435-5dff531f81e0-crio-socket\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.684896 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.684752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.785988 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.785901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f5fa97-3a75-482a-a435-5dff531f81e0-data-volume\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.785988 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.785946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.785988 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.785979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43f5fa97-3a75-482a-a435-5dff531f81e0-crio-socket\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.786168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.786062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.786168 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:16.786067 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:16.786168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.786106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdcx\" (UniqueName: \"kubernetes.io/projected/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-api-access-kjdcx\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.786168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.786106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43f5fa97-3a75-482a-a435-5dff531f81e0-crio-socket\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.786168 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:16.786142 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:17.286122474 +0000 UTC m=+85.660785631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:16.786341 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:16.786170 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:00:16.786341 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:16.786230 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:17.286212136 +0000 UTC m=+85.660875294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : configmap "kube-rbac-proxy" not found Jul 23 18:00:16.786341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.786287 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f5fa97-3a75-482a-a435-5dff531f81e0-data-volume\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:16.796927 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:16.796903 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdcx\" (UniqueName: \"kubernetes.io/projected/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-api-access-kjdcx\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:17.288979 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:17.288929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:17.289156 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:17.289016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:17.289156 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:17.289089 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:17.289156 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:17.289149 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:18.28913466 +0000 UTC m=+86.663797817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:17.289261 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:17.289100 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:00:17.289261 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:17.289217 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:18.289202707 +0000 UTC m=+86.663865882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : configmap "kube-rbac-proxy" not found Jul 23 18:00:18.296414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:18.296375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:18.296869 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:18.296449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:18.296869 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:18.296533 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:00:18.296869 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:18.296548 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:18.296869 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:18.296592 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:20.296577213 +0000 UTC m=+88.671240370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : configmap "kube-rbac-proxy" not found Jul 23 18:00:18.296869 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:18.296615 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:20.296597377 +0000 UTC m=+88.671260550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:18.516246 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:18.516217 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bvbtd_ef4c5948-06e9-4626-8545-e242dde01132/dns-node-resolver/0.log" Jul 23 18:00:19.115306 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:19.115280 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjkxn_ae691532-a868-4769-bde5-c93f7eb4e1d0/node-ca/0.log" Jul 23 18:00:19.305740 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:19.305659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:19.306200 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:19.305947 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:19.306200 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:19.306057 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.30602595 +0000 UTC m=+95.680689120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:19.407115 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:19.407082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:19.407285 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:19.407249 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:19.407338 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:19.407315 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.407300259 +0000 UTC m=+95.781963419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:20.115558 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:20.115527 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnj5r_0437e524-41b9-4b65-a9aa-6f24c560e36d/serve-healthcheck-canary/0.log" Jul 23 18:00:20.313233 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:20.313186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:20.313606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:20.313286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:20.313606 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:20.313326 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:20.313606 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:20.313364 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:00:20.313606 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:20.313389 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:24.31337416 +0000 UTC m=+92.688037318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:20.313606 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:20.313406 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:24.313395785 +0000 UTC m=+92.688058945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : configmap "kube-rbac-proxy" not found Jul 23 18:00:24.346033 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:24.345996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:24.346480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:24.346068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:24.346480 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:24.346139 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:00:24.346480 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:24.346161 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:24.346480 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:24.346218 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:32.346199001 +0000 UTC m=+100.720862162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : configmap "kube-rbac-proxy" not found Jul 23 18:00:24.346480 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:24.346232 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:32.346226896 +0000 UTC m=+100.720890053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:26.463719 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.463667 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-bspdr"] Jul 23 18:00:26.465712 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.465678 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.468048 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.468025 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 18:00:26.468841 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.468823 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 18:00:26.468890 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.468838 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r8q5k\"" Jul 23 18:00:26.468938 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.468840 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 18:00:26.468938 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.468823 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 18:00:26.475396 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.475375 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-bspdr"] Jul 23 18:00:26.561443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.561405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-key\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.561618 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.561459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-cabundle\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.561618 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.561484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpql\" (UniqueName: \"kubernetes.io/projected/3f247ea2-4c0d-4ab0-892b-60a128301b26-kube-api-access-wtpql\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.662553 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.662507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-cabundle\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.662658 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.662568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpql\" (UniqueName: \"kubernetes.io/projected/3f247ea2-4c0d-4ab0-892b-60a128301b26-kube-api-access-wtpql\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.662792 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.662771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-key\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.663280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.663264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-cabundle\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.665064 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.665046 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3f247ea2-4c0d-4ab0-892b-60a128301b26-signing-key\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.670846 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.670826 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpql\" (UniqueName: \"kubernetes.io/projected/3f247ea2-4c0d-4ab0-892b-60a128301b26-kube-api-access-wtpql\") pod \"service-ca-58598695f7-bspdr\" (UID: \"3f247ea2-4c0d-4ab0-892b-60a128301b26\") " pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.774532 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.774443 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-bspdr" Jul 23 18:00:26.896090 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:26.896057 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-bspdr"] Jul 23 18:00:26.899603 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:26.899575 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f247ea2_4c0d_4ab0_892b_60a128301b26.slice/crio-9441f07f2d8ba94b5b8ef18e4f1b8c2605894708ea6437496b01363586340ebe WatchSource:0}: Error finding container 9441f07f2d8ba94b5b8ef18e4f1b8c2605894708ea6437496b01363586340ebe: Status 404 returned error can't find the container with id 9441f07f2d8ba94b5b8ef18e4f1b8c2605894708ea6437496b01363586340ebe Jul 23 18:00:27.368644 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:27.368612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:27.368866 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:27.368795 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 18:00:27.368936 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:27.368867 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls podName:faa76306-034d-4627-ae32-0ca13712c750 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:43.36884818 +0000 UTC m=+111.743511356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jwspg" (UID: "faa76306-034d-4627-ae32-0ca13712c750") : secret "samples-operator-tls" not found Jul 23 18:00:27.469490 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:27.469453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:27.469886 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:27.469630 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:27.469886 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:27.469715 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls podName:febde259-4b55-421f-a4ec-1b4b78af5eac nodeName:}" failed. No retries permitted until 2026-07-23 18:00:43.469675364 +0000 UTC m=+111.844338540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-flk6k" (UID: "febde259-4b55-421f-a4ec-1b4b78af5eac") : secret "cluster-monitoring-operator-tls" not found Jul 23 18:00:27.552931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:27.552895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-bspdr" event={"ID":"3f247ea2-4c0d-4ab0-892b-60a128301b26","Type":"ContainerStarted","Data":"9441f07f2d8ba94b5b8ef18e4f1b8c2605894708ea6437496b01363586340ebe"} Jul 23 18:00:29.559086 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:29.559001 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-bspdr" event={"ID":"3f247ea2-4c0d-4ab0-892b-60a128301b26","Type":"ContainerStarted","Data":"bd8d2e816ce5a45da8ee9d4796cff2335f6b202a945c305e5b25bf12c653e04d"} Jul 23 18:00:29.578129 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:29.578082 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-bspdr" podStartSLOduration=1.317742677 podStartE2EDuration="3.578066499s" totalImagesPullingTime="2.260323822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:29.576581711 +0000 UTC m=+97.951244890" watchObservedRunningTime="2026-07-23 18:00:29.578066499 +0000 UTC m=+97.952729656" Jul 23 18:00:30.998770 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:30.998732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 18:00:30.999095 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:30.998915 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:00:30.999095 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:30.998994 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls podName:f2e796a2-1a03-48bf-85b6-e8a1f549b5fd nodeName:}" failed. No retries permitted until 2026-07-23 18:01:34.998973192 +0000 UTC m=+163.373636361 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls") pod "dns-default-j7q2q" (UID: "f2e796a2-1a03-48bf-85b6-e8a1f549b5fd") : secret "dns-default-metrics-tls" not found Jul 23 18:00:31.807804 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:31.807761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") pod \"image-registry-659fb474bb-62jbx\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 18:00:31.808006 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:31.807923 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:00:31.808006 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:31.807946 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659fb474bb-62jbx: secret "image-registry-tls" not found Jul 23 18:00:31.808126 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:31.808014 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls podName:cfccd801-4b47-4f7e-bbac-92e67e3e18de nodeName:}" failed. No retries permitted until 2026-07-23 18:01:35.807991391 +0000 UTC m=+164.182654564 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls") pod "image-registry-659fb474bb-62jbx" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de") : secret "image-registry-tls" not found Jul 23 18:00:32.410699 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:32.410649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:32.411071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:32.410753 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:32.411071 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:32.410851 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:00:32.411071 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:00:32.410903 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls podName:43f5fa97-3a75-482a-a435-5dff531f81e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:48.41088991 +0000 UTC m=+116.785553068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxkrb" (UID: "43f5fa97-3a75-482a-a435-5dff531f81e0") : secret "insights-runtime-extractor-tls" not found Jul 23 18:00:32.411177 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:32.411116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43f5fa97-3a75-482a-a435-5dff531f81e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:34.498198 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:34.498165 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s789k" Jul 23 18:00:43.399343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.399302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:43.401655 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.401629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/faa76306-034d-4627-ae32-0ca13712c750-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jwspg\" (UID: \"faa76306-034d-4627-ae32-0ca13712c750\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:43.420497 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.420474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" Jul 23 18:00:43.500256 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.500225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:43.502453 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.502427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/febde259-4b55-421f-a4ec-1b4b78af5eac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-flk6k\" (UID: \"febde259-4b55-421f-a4ec-1b4b78af5eac\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:43.509351 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.509323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" Jul 23 18:00:43.542373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.542344 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg"] Jul 23 18:00:43.597378 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.597353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" event={"ID":"faa76306-034d-4627-ae32-0ca13712c750","Type":"ContainerStarted","Data":"5769af237df0033ae42422155f849169c11685391afffbf1f9d7593d460fe015"} Jul 23 18:00:43.635416 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:43.635386 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k"] Jul 23 18:00:43.638523 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:43.638493 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebde259_4b55_421f_a4ec_1b4b78af5eac.slice/crio-bb2a2221b733fa061a848b9d55f5c2886ae78ecddfa3e5c96ff6f1c13a5983bd WatchSource:0}: Error finding container bb2a2221b733fa061a848b9d55f5c2886ae78ecddfa3e5c96ff6f1c13a5983bd: Status 404 returned error can't find the container with id bb2a2221b733fa061a848b9d55f5c2886ae78ecddfa3e5c96ff6f1c13a5983bd Jul 23 18:00:44.602071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:44.602001 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" event={"ID":"febde259-4b55-421f-a4ec-1b4b78af5eac","Type":"ContainerStarted","Data":"bb2a2221b733fa061a848b9d55f5c2886ae78ecddfa3e5c96ff6f1c13a5983bd"} Jul 23 18:00:46.498798 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.498760 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf"] Jul 23 18:00:46.500536 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.500519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.539402 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.539368 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 18:00:46.539402 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.539390 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 18:00:46.539402 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.539397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mlrr2\"" Jul 23 18:00:46.539623 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.539440 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 18:00:46.539623 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.539390 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 18:00:46.545157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.545133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf"] Jul 23 18:00:46.610851 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.610797 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" event={"ID":"faa76306-034d-4627-ae32-0ca13712c750","Type":"ContainerStarted","Data":"6080c3e446e2bf6256ef105ae0e42ed0bce1abe82be145b74025aecae5084329"} Jul 23 18:00:46.611014 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.610915 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" event={"ID":"faa76306-034d-4627-ae32-0ca13712c750","Type":"ContainerStarted","Data":"d01b0585f6fb8a4c8692b94bf7b1c69837dc0f2cf5bc3575a500af704edc1099"} Jul 23 18:00:46.616252 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.616225 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-lx2mz"] Jul 23 18:00:46.618178 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.618156 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" event={"ID":"febde259-4b55-421f-a4ec-1b4b78af5eac","Type":"ContainerStarted","Data":"1efbe50c295b22ca8af52df4e3f359ec2a51761dbebf4c563f7e22e4305eae39"} Jul 23 18:00:46.618271 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.618227 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:00:46.622634 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.622613 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 18:00:46.622764 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.622658 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 18:00:46.623060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.623036 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kfnm6\"" Jul 23 18:00:46.625947 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.625930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2869238a-047a-4942-a477-6dea57a13553-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.626021 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.625978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw47v\" (UniqueName: \"kubernetes.io/projected/2869238a-047a-4942-a477-6dea57a13553-kube-api-access-nw47v\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.659397 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.659369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-lx2mz"] Jul 23 18:00:46.662377 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.662331 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jwspg" podStartSLOduration=33.472470705 podStartE2EDuration="35.662314183s" totalImagesPullingTime="2.189843478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:46.662006592 +0000 UTC m=+115.036669784" watchObservedRunningTime="2026-07-23 18:00:46.662314183 +0000 UTC m=+115.036977363" Jul 23 18:00:46.697926 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.697883 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-flk6k" podStartSLOduration=33.560537079 podStartE2EDuration="35.697869382s" totalImagesPullingTime="2.137332303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:46.697432894 +0000 UTC m=+115.072096063" watchObservedRunningTime="2026-07-23 18:00:46.697869382 +0000 UTC m=+115.072532561" Jul 23 18:00:46.714651 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.714623 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf"] Jul 23 18:00:46.716818 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.716801 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:46.721613 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.721591 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zh5t7\"" Jul 23 18:00:46.721729 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.721658 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 18:00:46.726884 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.726864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw47v\" (UniqueName: \"kubernetes.io/projected/2869238a-047a-4942-a477-6dea57a13553-kube-api-access-nw47v\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.727012 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.726995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhvf\" (UniqueName: \"kubernetes.io/projected/06d151cf-b03b-45af-a6bb-beaa21499166-kube-api-access-bdhvf\") pod \"downloads-5bc67dcd46-lx2mz\" (UID: \"06d151cf-b03b-45af-a6bb-beaa21499166\") " pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:00:46.727078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.727062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2869238a-047a-4942-a477-6dea57a13553-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.729560 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.729532 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2869238a-047a-4942-a477-6dea57a13553-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.735526 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.735505 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf"] Jul 23 18:00:46.745327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.745309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw47v\" (UniqueName: \"kubernetes.io/projected/2869238a-047a-4942-a477-6dea57a13553-kube-api-access-nw47v\") pod \"managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf\" (UID: \"2869238a-047a-4942-a477-6dea57a13553\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.820266 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.820190 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" Jul 23 18:00:46.828323 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.828292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhvf\" (UniqueName: \"kubernetes.io/projected/06d151cf-b03b-45af-a6bb-beaa21499166-kube-api-access-bdhvf\") pod \"downloads-5bc67dcd46-lx2mz\" (UID: \"06d151cf-b03b-45af-a6bb-beaa21499166\") " pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:00:46.828427 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.828333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af882abd-0c56-4f18-bd04-adfa89e66794-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6qncf\" (UID: \"af882abd-0c56-4f18-bd04-adfa89e66794\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:46.837805 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.837781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhvf\" (UniqueName: \"kubernetes.io/projected/06d151cf-b03b-45af-a6bb-beaa21499166-kube-api-access-bdhvf\") pod \"downloads-5bc67dcd46-lx2mz\" (UID: \"06d151cf-b03b-45af-a6bb-beaa21499166\") " pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:00:46.926691 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.926652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:00:46.928933 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.928905 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af882abd-0c56-4f18-bd04-adfa89e66794-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6qncf\" (UID: \"af882abd-0c56-4f18-bd04-adfa89e66794\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:46.931217 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.931187 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af882abd-0c56-4f18-bd04-adfa89e66794-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6qncf\" (UID: \"af882abd-0c56-4f18-bd04-adfa89e66794\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:46.941981 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:46.941956 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf"] Jul 23 18:00:46.944625 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:46.944602 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2869238a_047a_4942_a477_6dea57a13553.slice/crio-1dbb9ab27fc038483afe09633449f6c22097e98d4106da7df8bb534ad0f38713 WatchSource:0}: Error finding container 1dbb9ab27fc038483afe09633449f6c22097e98d4106da7df8bb534ad0f38713: Status 404 returned error can't find the container with id 1dbb9ab27fc038483afe09633449f6c22097e98d4106da7df8bb534ad0f38713 Jul 23 18:00:47.025141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.025111 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:47.069579 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.069545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-lx2mz"] Jul 23 18:00:47.072706 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:47.072660 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d151cf_b03b_45af_a6bb_beaa21499166.slice/crio-285b87529be35969be714e83dfd86aa394800c71276ea815d9b75ab769426b56 WatchSource:0}: Error finding container 285b87529be35969be714e83dfd86aa394800c71276ea815d9b75ab769426b56: Status 404 returned error can't find the container with id 285b87529be35969be714e83dfd86aa394800c71276ea815d9b75ab769426b56 Jul 23 18:00:47.179007 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.178973 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf"] Jul 23 18:00:47.181988 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:47.181960 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf882abd_0c56_4f18_bd04_adfa89e66794.slice/crio-3f1cb2bb254e8f917220d9b87dde40ce90062a71b9763b69651bc347fdf2b336 WatchSource:0}: Error finding container 3f1cb2bb254e8f917220d9b87dde40ce90062a71b9763b69651bc347fdf2b336: Status 404 returned error can't find the container with id 3f1cb2bb254e8f917220d9b87dde40ce90062a71b9763b69651bc347fdf2b336 Jul 23 18:00:47.625902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.625836 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" event={"ID":"af882abd-0c56-4f18-bd04-adfa89e66794","Type":"ContainerStarted","Data":"3f1cb2bb254e8f917220d9b87dde40ce90062a71b9763b69651bc347fdf2b336"} Jul 23 18:00:47.627193 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.627155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" event={"ID":"2869238a-047a-4942-a477-6dea57a13553","Type":"ContainerStarted","Data":"1dbb9ab27fc038483afe09633449f6c22097e98d4106da7df8bb534ad0f38713"} Jul 23 18:00:47.628504 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:47.628477 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-lx2mz" event={"ID":"06d151cf-b03b-45af-a6bb-beaa21499166","Type":"ContainerStarted","Data":"285b87529be35969be714e83dfd86aa394800c71276ea815d9b75ab769426b56"} Jul 23 18:00:48.443304 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.443270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:48.445878 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.445847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43f5fa97-3a75-482a-a435-5dff531f81e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxkrb\" (UID: \"43f5fa97-3a75-482a-a435-5dff531f81e0\") " pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:48.634025 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.633982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" event={"ID":"af882abd-0c56-4f18-bd04-adfa89e66794","Type":"ContainerStarted","Data":"427c4f14a013113905c729b150d7486babfaf3c14b59485ea3ad017a87c055fe"} Jul 23 18:00:48.634497 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.634282 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:48.640257 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.640231 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" Jul 23 18:00:48.654799 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.654739 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6qncf" podStartSLOduration=1.520044884 podStartE2EDuration="2.654718865s" totalImagesPullingTime="1.134673981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:48.654125615 +0000 UTC m=+117.028788791" watchObservedRunningTime="2026-07-23 18:00:48.654718865 +0000 UTC m=+117.029382044" Jul 23 18:00:48.658068 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.658042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxkrb" Jul 23 18:00:48.821132 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:48.821101 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxkrb"] Jul 23 18:00:48.822459 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:48.822427 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f5fa97_3a75_482a_a435_5dff531f81e0.slice/crio-d37c7c918f526af5064aff0a8e5fe6e8d26831c8c99dc367b356be8bf924b3ee WatchSource:0}: Error finding container d37c7c918f526af5064aff0a8e5fe6e8d26831c8c99dc367b356be8bf924b3ee: Status 404 returned error can't find the container with id d37c7c918f526af5064aff0a8e5fe6e8d26831c8c99dc367b356be8bf924b3ee Jul 23 18:00:49.226669 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.226636 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:00:49.229323 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.229303 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.232000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.231973 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 18:00:49.232118 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.232033 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 18:00:49.232118 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.232045 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 18:00:49.232118 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.232038 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 18:00:49.232922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.232883 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 18:00:49.232922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.232908 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wddp2\"" Jul 23 18:00:49.241786 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.241765 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:00:49.351794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.351751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.351975 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.351807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.351975 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.351910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.352051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.351992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.352051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.352011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.352051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.352032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rljx\" (UniqueName: \"kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452702 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452702 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rljx\" (UniqueName: \"kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.452931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.452909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.453425 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.453394 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.453745 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.453673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.453947 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.453915 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.455576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.455553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.455992 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.455945 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.462601 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.462557 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rljx\" (UniqueName: \"kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx\") pod \"console-7b99b9575-zxw9g\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.540487 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.540419 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:49.639277 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.639241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxkrb" event={"ID":"43f5fa97-3a75-482a-a435-5dff531f81e0","Type":"ContainerStarted","Data":"d37c7c918f526af5064aff0a8e5fe6e8d26831c8c99dc367b356be8bf924b3ee"} Jul 23 18:00:49.689076 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:49.689042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:00:49.692670 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:00:49.692643 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e17f35_2f9c_4891_b603_7920937d8bbf.slice/crio-692df2f2d226b696a9be7ea116987d281ea55827a3f39071fbc56746dd881b16 WatchSource:0}: Error finding container 692df2f2d226b696a9be7ea116987d281ea55827a3f39071fbc56746dd881b16: Status 404 returned error can't find the container with id 692df2f2d226b696a9be7ea116987d281ea55827a3f39071fbc56746dd881b16 Jul 23 18:00:50.647166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:50.647124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99b9575-zxw9g" event={"ID":"14e17f35-2f9c-4891-b603-7920937d8bbf","Type":"ContainerStarted","Data":"692df2f2d226b696a9be7ea116987d281ea55827a3f39071fbc56746dd881b16"} Jul 23 18:00:57.684770 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:57.684630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxkrb" event={"ID":"43f5fa97-3a75-482a-a435-5dff531f81e0","Type":"ContainerStarted","Data":"0395cfa7b1d0c687e0665d6c10fa111cbea4c7e0f1ed151e7f7b92a962a12874"} Jul 23 18:00:57.688964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:57.688930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99b9575-zxw9g" event={"ID":"14e17f35-2f9c-4891-b603-7920937d8bbf","Type":"ContainerStarted","Data":"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038"} Jul 23 18:00:57.692174 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:57.691602 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" event={"ID":"2869238a-047a-4942-a477-6dea57a13553","Type":"ContainerStarted","Data":"28362659c3a00b67f27292ea3f8d3e89d98713181ee334af801692c6eaebd238"} Jul 23 18:00:57.712898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:57.712716 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b99b9575-zxw9g" podStartSLOduration=1.890268142 podStartE2EDuration="8.712673265s" totalImagesPullingTime="6.822405123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:57.710131335 +0000 UTC m=+126.084794514" watchObservedRunningTime="2026-07-23 18:00:57.712673265 +0000 UTC m=+126.087336446" Jul 23 18:00:57.731735 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:57.731288 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b45cb7bf4-2rqjf" podStartSLOduration=2.160392972 podStartE2EDuration="11.731270826s" totalImagesPullingTime="9.570877854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:57.730171633 +0000 UTC m=+126.104834813" watchObservedRunningTime="2026-07-23 18:00:57.731270826 +0000 UTC m=+126.105934006" Jul 23 18:00:58.890533 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.890504 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:00:58.896641 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.896619 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.924865 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.924840 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:00:58.926109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.926036 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 18:00:58.934955 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.934932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935048 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.934964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935048 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.935002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hkq\" (UniqueName: \"kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.935045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.935108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.935146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:58.935234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:58.935177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036115 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hkq\" (UniqueName: \"kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036465 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036465 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.036465 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.036394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.037165 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.037134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.037292 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.037168 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.037292 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.037174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.037292 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.037276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.038968 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.038943 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.039068 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.039051 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.049284 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.049266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hkq\" (UniqueName: \"kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq\") pod \"console-75766b8f9c-xl98s\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.207165 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.207072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:00:59.541972 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.541891 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:59.542131 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.542001 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:00:59.543475 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.543424 2568 patch_prober.go:28] interesting pod/console-7b99b9575-zxw9g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Jul 23 18:00:59.543614 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:00:59.543479 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7b99b9575-zxw9g" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" Jul 23 18:01:00.340671 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.340633 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h4zgm"] Jul 23 18:01:00.343731 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.343710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.347120 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.347100 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 18:01:00.347774 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.347353 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 18:01:00.347774 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.347584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 18:01:00.347774 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.347676 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ghbr\"" Jul 23 18:01:00.348624 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.348596 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 18:01:00.449203 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-root\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449376 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-wtmp\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449376 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-sys\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449373 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-tls\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449466 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-textfile\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449668 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-metrics-client-ca\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.449668 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.449554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qtn\" (UniqueName: \"kubernetes.io/projected/d777d537-fec3-4930-8c4f-697074fdc4b6-kube-api-access-l4qtn\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551014 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.550972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-textfile\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-metrics-client-ca\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551353 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qtn\" (UniqueName: \"kubernetes.io/projected/d777d537-fec3-4930-8c4f-697074fdc4b6-kube-api-access-l4qtn\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551353 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-root\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551353 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-wtmp\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-sys\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-tls\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551651 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-root\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551777 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551756 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551756 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d777d537-fec3-4930-8c4f-697074fdc4b6-metrics-client-ca\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551804 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-sys\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.551843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.551832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-wtmp\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.552113 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.552047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-textfile\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.553895 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.553870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.554074 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.554052 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d777d537-fec3-4930-8c4f-697074fdc4b6-node-exporter-tls\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.569337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.569289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qtn\" (UniqueName: \"kubernetes.io/projected/d777d537-fec3-4930-8c4f-697074fdc4b6-kube-api-access-l4qtn\") pod \"node-exporter-h4zgm\" (UID: \"d777d537-fec3-4930-8c4f-697074fdc4b6\") " pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.655794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.655758 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4zgm" Jul 23 18:01:00.940013 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.939922 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:01:00.945240 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.945216 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:00.951000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.950979 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 18:01:00.951186 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.951038 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 18:01:00.951337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.951128 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 18:01:00.952254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.952228 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 18:01:00.952254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.952245 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 18:01:00.952480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.952462 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 18:01:00.952574 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.952517 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pmb2r\"" Jul 23 18:01:00.954392 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.954372 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 18:01:00.954623 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.954606 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 18:01:00.954956 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.954935 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 18:01:00.975369 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:00.975338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:01:01.056567 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056756 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056756 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056756 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.056912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056855 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.056961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.057003 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.057029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057246 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.057063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.057246 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.057088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqjf\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158413 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158413 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqjf\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158996 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158996 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158996 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158996 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158788 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.158996 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.158833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.160011 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.159645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.160169 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.160094 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.160395 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.160351 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.162429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.162256 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.162429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.162386 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.162429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.162409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.162637 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.162458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.163050 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.163017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.163605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.163580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.163787 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.163766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.164228 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.164203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.164359 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.164335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.172547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.172523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqjf\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf\") pod \"alertmanager-main-0\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.259395 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.259311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:01:01.965278 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.965236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 18:01:01.968079 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:01.968053 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdc93a0-ac64-4d07-ab67-96e3e6339cff-metrics-certs\") pod \"network-metrics-daemon-vlpnb\" (UID: \"5bdc93a0-ac64-4d07-ab67-96e3e6339cff\") " pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 18:01:02.262307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:02.262216 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hv4kc\"" Jul 23 18:01:02.269488 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:02.269460 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlpnb" Jul 23 18:01:04.694388 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.694349 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fb598c4fd-z5xht"] Jul 23 18:01:04.697544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.697517 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.701571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 18:01:04.701571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701325 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 18:01:04.701571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701360 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6cku2fd9at2kq\"" Jul 23 18:01:04.701571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701293 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 18:01:04.701875 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701628 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 18:01:04.701875 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.701659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bpl5h\"" Jul 23 18:01:04.713205 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.713140 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb598c4fd-z5xht"] Jul 23 18:01:04.791808 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-client-certs\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.791973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-metrics-server-audit-profiles\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.791973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-client-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.791973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be25b4be-5347-41ed-8205-f1bc60e0ee7e-audit-log\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.791973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.792170 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.791988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-tls\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.792170 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.792109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7742\" (UniqueName: \"kubernetes.io/projected/be25b4be-5347-41ed-8205-f1bc60e0ee7e-kube-api-access-t7742\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.892976 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.892934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7742\" (UniqueName: \"kubernetes.io/projected/be25b4be-5347-41ed-8205-f1bc60e0ee7e-kube-api-access-t7742\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-client-certs\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-metrics-server-audit-profiles\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-client-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be25b4be-5347-41ed-8205-f1bc60e0ee7e-audit-log\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-tls\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.893938 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893912 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be25b4be-5347-41ed-8205-f1bc60e0ee7e-audit-log\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.894083 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.893914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.894083 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.894047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be25b4be-5347-41ed-8205-f1bc60e0ee7e-metrics-server-audit-profiles\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.895797 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.895776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-client-ca-bundle\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.896015 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.895993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-tls\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.896092 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.896051 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be25b4be-5347-41ed-8205-f1bc60e0ee7e-secret-metrics-server-client-certs\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:04.905299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:04.905258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7742\" (UniqueName: \"kubernetes.io/projected/be25b4be-5347-41ed-8205-f1bc60e0ee7e-kube-api-access-t7742\") pod \"metrics-server-6fb598c4fd-z5xht\" (UID: \"be25b4be-5347-41ed-8205-f1bc60e0ee7e\") " pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:05.009265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.009175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:05.418621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.418582 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:01:05.457555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.457525 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:01:05.465084 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.465057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.473146 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.473116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:01:05.499040 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499184 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499052 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499184 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499184 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499332 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499332 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5clk\" (UniqueName: \"kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.499332 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.499287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5clk\" (UniqueName: \"kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600337 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.600714 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.600470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.601227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.601127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.601227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.601155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.601227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.601135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.601433 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.601393 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.603667 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.603629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.603859 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.603843 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.609563 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.609534 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5clk\" (UniqueName: \"kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk\") pod \"console-7666b4b565-6chfh\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:05.776628 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:05.776537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:06.163478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.163446 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:01:06.220241 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.220208 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:01:06.220405 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.220359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223123 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ldvpn\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223133 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223125 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223192 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223136 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223306 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223335 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 18:01:06.223414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223338 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 18:01:06.223903 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.223598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39qesgq6uev3p\"" Jul 23 18:01:06.224473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.224453 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 18:01:06.224473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.224466 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 18:01:06.224725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.224708 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 18:01:06.224900 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.224885 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 18:01:06.224965 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.224891 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 18:01:06.229324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.228443 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 18:01:06.234009 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.233969 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 18:01:06.307948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.307916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.307978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308155 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.308803 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.308549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw74d\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.409931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.409882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.409931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.409931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.409967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.409996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw74d\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410401 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.410502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410495 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.411008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.411008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.411008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.410914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.411552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.411448 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.412701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.412513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.412849 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.412754 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.412991 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.412968 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.413579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.413645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.413799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.414036 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.414074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414738 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.414649 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.414950 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.414907 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.415295 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.415271 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.416376 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.416352 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.416606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.416563 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.417194 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.417157 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.417285 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.417245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.417285 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.417251 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.423555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.423529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw74d\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d\") pod \"prometheus-k8s-0\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:06.532603 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:06.532569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:08.241598 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.241563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd777d537_fec3_4930_8c4f_697074fdc4b6.slice/crio-26e0e3c0869476f6c7c04bd46236017357c8d7780221cbac1d533fd8187bc148 WatchSource:0}: Error finding container 26e0e3c0869476f6c7c04bd46236017357c8d7780221cbac1d533fd8187bc148: Status 404 returned error can't find the container with id 26e0e3c0869476f6c7c04bd46236017357c8d7780221cbac1d533fd8187bc148 Jul 23 18:01:08.441843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.441398 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:01:08.445088 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.444993 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e20eab_9d6b_4494_a7f5_0902c3452a7f.slice/crio-d9e634c74c90e7e5e4b74e574acc814bc18340d370723494b5c79397e7c4aaa5 WatchSource:0}: Error finding container d9e634c74c90e7e5e4b74e574acc814bc18340d370723494b5c79397e7c4aaa5: Status 404 returned error can't find the container with id d9e634c74c90e7e5e4b74e574acc814bc18340d370723494b5c79397e7c4aaa5 Jul 23 18:01:08.461640 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.461539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:01:08.464977 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.464913 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded27d456_7901_49d7_8ac7_60f7a1be4771.slice/crio-bad9368881807a4a32366dc2a00293130d8c2780c76c5f2e7a23b774fa5faaa1 WatchSource:0}: Error finding container bad9368881807a4a32366dc2a00293130d8c2780c76c5f2e7a23b774fa5faaa1: Status 404 returned error can't find the container with id bad9368881807a4a32366dc2a00293130d8c2780c76c5f2e7a23b774fa5faaa1 Jul 23 18:01:08.682675 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.682594 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb598c4fd-z5xht"] Jul 23 18:01:08.687531 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.687503 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:01:08.690697 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.690645 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5928ac32_439b_49c0_8004_212f8c32ff3d.slice/crio-127f7948fc5b85e6643d80bf5b17f66347e6540dc7111da01685624409c8bd0b WatchSource:0}: Error finding container 127f7948fc5b85e6643d80bf5b17f66347e6540dc7111da01685624409c8bd0b: Status 404 returned error can't find the container with id 127f7948fc5b85e6643d80bf5b17f66347e6540dc7111da01685624409c8bd0b Jul 23 18:01:08.724210 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.724109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vlpnb"] Jul 23 18:01:08.729574 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.729545 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdc93a0_ac64_4d07_ab67_96e3e6339cff.slice/crio-454725da0563ed4c780bd6b4b3ad493b9903694369e398f19d8b78648a7aed88 WatchSource:0}: Error finding container 454725da0563ed4c780bd6b4b3ad493b9903694369e398f19d8b78648a7aed88: Status 404 returned error can't find the container with id 454725da0563ed4c780bd6b4b3ad493b9903694369e398f19d8b78648a7aed88 Jul 23 18:01:08.734764 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.734152 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:01:08.740114 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.740084 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"bad9368881807a4a32366dc2a00293130d8c2780c76c5f2e7a23b774fa5faaa1"} Jul 23 18:01:08.740528 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:08.740468 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a854d20_ddbf_457d_8016_c3379fffce9a.slice/crio-ead16eb5db2b03fd256340c7150a39301208dd25179e9501bf3b90ffe63d4f82 WatchSource:0}: Error finding container ead16eb5db2b03fd256340c7150a39301208dd25179e9501bf3b90ffe63d4f82: Status 404 returned error can't find the container with id ead16eb5db2b03fd256340c7150a39301208dd25179e9501bf3b90ffe63d4f82 Jul 23 18:01:08.746054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.745950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-lx2mz" event={"ID":"06d151cf-b03b-45af-a6bb-beaa21499166","Type":"ContainerStarted","Data":"d07d8b1b95e479249af9c0311def94eaddacd2eac82f7c3d45bd0444ea16a5a5"} Jul 23 18:01:08.746054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.746012 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:01:08.752041 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.751964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxkrb" event={"ID":"43f5fa97-3a75-482a-a435-5dff531f81e0","Type":"ContainerStarted","Data":"eacb8c93c58a48140d62b7d9a54b3a2e8574eb1d7dbbd8584821658b61d72a2c"} Jul 23 18:01:08.754367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.754284 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4zgm" event={"ID":"d777d537-fec3-4930-8c4f-697074fdc4b6","Type":"ContainerStarted","Data":"26e0e3c0869476f6c7c04bd46236017357c8d7780221cbac1d533fd8187bc148"} Jul 23 18:01:08.757027 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.756908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7666b4b565-6chfh" event={"ID":"5928ac32-439b-49c0-8004-212f8c32ff3d","Type":"ContainerStarted","Data":"127f7948fc5b85e6643d80bf5b17f66347e6540dc7111da01685624409c8bd0b"} Jul 23 18:01:08.757615 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.757574 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-lx2mz" Jul 23 18:01:08.759532 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.759492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b8f9c-xl98s" event={"ID":"55e20eab-9d6b-4494-a7f5-0902c3452a7f","Type":"ContainerStarted","Data":"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f"} Jul 23 18:01:08.759532 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.759527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b8f9c-xl98s" event={"ID":"55e20eab-9d6b-4494-a7f5-0902c3452a7f","Type":"ContainerStarted","Data":"d9e634c74c90e7e5e4b74e574acc814bc18340d370723494b5c79397e7c4aaa5"} Jul 23 18:01:08.761180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.761153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" event={"ID":"be25b4be-5347-41ed-8205-f1bc60e0ee7e","Type":"ContainerStarted","Data":"d62e87c1a1dd72f7a2b81289d4b586b2448258a5352652d7e48c56b79ec489f7"} Jul 23 18:01:08.771727 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.771105 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-lx2mz" podStartSLOduration=1.559271924 podStartE2EDuration="22.771093446s" totalImagesPullingTime="21.211821522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:08.770741635 +0000 UTC m=+137.145404814" watchObservedRunningTime="2026-07-23 18:01:08.771093446 +0000 UTC m=+137.145756624" Jul 23 18:01:08.792357 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:08.792300 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75766b8f9c-xl98s" podStartSLOduration=10.79228289 podStartE2EDuration="10.79228289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:08.792222133 +0000 UTC m=+137.166885326" watchObservedRunningTime="2026-07-23 18:01:08.79228289 +0000 UTC m=+137.166946070" Jul 23 18:01:09.209201 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.207224 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:09.209201 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.209166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:09.215810 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.215786 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:09.770951 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.770088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7666b4b565-6chfh" event={"ID":"5928ac32-439b-49c0-8004-212f8c32ff3d","Type":"ContainerStarted","Data":"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a"} Jul 23 18:01:09.773283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.773178 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"ead16eb5db2b03fd256340c7150a39301208dd25179e9501bf3b90ffe63d4f82"} Jul 23 18:01:09.776648 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.776553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlpnb" event={"ID":"5bdc93a0-ac64-4d07-ab67-96e3e6339cff","Type":"ContainerStarted","Data":"454725da0563ed4c780bd6b4b3ad493b9903694369e398f19d8b78648a7aed88"} Jul 23 18:01:09.782897 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.782835 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:09.818956 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:09.818889 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7666b4b565-6chfh" podStartSLOduration=4.818867797 podStartE2EDuration="4.818867797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:09.791772753 +0000 UTC m=+138.166435931" watchObservedRunningTime="2026-07-23 18:01:09.818867797 +0000 UTC m=+138.193530977" Jul 23 18:01:10.725369 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.725327 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659fb474bb-62jbx"] Jul 23 18:01:10.725795 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:01:10.725726 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" podUID="cfccd801-4b47-4f7e-bbac-92e67e3e18de" Jul 23 18:01:10.790799 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.788871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 18:01:10.822855 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.822824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 18:01:10.962042 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.961985 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962042 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962039 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962407 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962077 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962407 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962125 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962407 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962157 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962407 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962211 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmc52\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.962407 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.962246 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") pod \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\" (UID: \"cfccd801-4b47-4f7e-bbac-92e67e3e18de\") " Jul 23 18:01:10.964216 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.964148 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:01:10.964359 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.964268 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:10.966478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.965943 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:10.966478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.966118 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:01:10.969925 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.969888 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:10.977959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.972223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52" (OuterVolumeSpecName: "kube-api-access-mmc52") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "kube-api-access-mmc52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:01:10.977959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:10.976551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cfccd801-4b47-4f7e-bbac-92e67e3e18de" (UID: "cfccd801-4b47-4f7e-bbac-92e67e3e18de"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:11.063645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063602 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmc52\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-kube-api-access-mmc52\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063639 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-trusted-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063915 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063655 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-image-registry-private-configuration\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063915 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063669 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cfccd801-4b47-4f7e-bbac-92e67e3e18de-installation-pull-secrets\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063915 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063699 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cfccd801-4b47-4f7e-bbac-92e67e3e18de-ca-trust-extracted\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063915 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063713 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-bound-sa-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.063915 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.063726 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-certificates\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:11.793111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.793075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659fb474bb-62jbx" Jul 23 18:01:11.832322 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.832290 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659fb474bb-62jbx"] Jul 23 18:01:11.841319 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.841278 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-659fb474bb-62jbx"] Jul 23 18:01:11.975513 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:11.975423 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cfccd801-4b47-4f7e-bbac-92e67e3e18de-registry-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:12.153089 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:12.153051 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfccd801-4b47-4f7e-bbac-92e67e3e18de" path="/var/lib/kubelet/pods/cfccd801-4b47-4f7e-bbac-92e67e3e18de/volumes" Jul 23 18:01:14.813226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:14.813193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4zgm" event={"ID":"d777d537-fec3-4930-8c4f-697074fdc4b6","Type":"ContainerStarted","Data":"25781162e58db18024a626c4c6a00fab5b83ea3439f9b368f8d1046b8d41f236"} Jul 23 18:01:15.777356 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.777319 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:15.777540 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.777402 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:15.782648 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.782624 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:15.819702 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.819644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxkrb" event={"ID":"43f5fa97-3a75-482a-a435-5dff531f81e0","Type":"ContainerStarted","Data":"c0e804d8bc9e71ab7bc3a72537ad2f4c199552a9f539524178036760aa569080"} Jul 23 18:01:15.821766 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.821734 2568 generic.go:356] "Generic (PLEG): container finished" podID="d777d537-fec3-4930-8c4f-697074fdc4b6" containerID="25781162e58db18024a626c4c6a00fab5b83ea3439f9b368f8d1046b8d41f236" exitCode=0 Jul 23 18:01:15.821888 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.821798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4zgm" event={"ID":"d777d537-fec3-4930-8c4f-697074fdc4b6","Type":"ContainerDied","Data":"25781162e58db18024a626c4c6a00fab5b83ea3439f9b368f8d1046b8d41f236"} Jul 23 18:01:15.823668 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.823645 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" exitCode=0 Jul 23 18:01:15.823800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.823782 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} Jul 23 18:01:15.825938 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.825913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" event={"ID":"be25b4be-5347-41ed-8205-f1bc60e0ee7e","Type":"ContainerStarted","Data":"6398152796fbeabd2bb42ae8422d3a89d9c8b4f95675c0fc4dd958a8f451292c"} Jul 23 18:01:15.828004 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.827977 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea" exitCode=0 Jul 23 18:01:15.828106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.828067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea"} Jul 23 18:01:15.834149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.834119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlpnb" event={"ID":"5bdc93a0-ac64-4d07-ab67-96e3e6339cff","Type":"ContainerStarted","Data":"01ef0bdac73fec0c5f2d8eb7f83f9a9238451c0cdfc9a7cc553baed9f4b516ff"} Jul 23 18:01:15.834267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.834175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlpnb" event={"ID":"5bdc93a0-ac64-4d07-ab67-96e3e6339cff","Type":"ContainerStarted","Data":"276e64c6e5d9e4e841c363e9335a90d74d8c0d7df4d2c0fad4d17327cc7f97c7"} Jul 23 18:01:15.838734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.838670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hxkrb" podStartSLOduration=34.139668235 podStartE2EDuration="59.838653451s" totalImagesPullingTime="25.698985216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:00:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:15.837183125 +0000 UTC m=+144.211846305" watchObservedRunningTime="2026-07-23 18:01:15.838653451 +0000 UTC m=+144.213316632" Jul 23 18:01:15.839312 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.839291 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:01:15.852851 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.852806 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vlpnb" podStartSLOduration=137.930635838 podStartE2EDuration="2m23.852787912s" totalImagesPullingTime="5.922152074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:15.85210184 +0000 UTC m=+144.226765020" watchObservedRunningTime="2026-07-23 18:01:15.852787912 +0000 UTC m=+144.227451093" Jul 23 18:01:15.871506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:15.871310 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" podStartSLOduration=5.902114207 podStartE2EDuration="11.871293194s" totalImagesPullingTime="5.969178987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:15.86915163 +0000 UTC m=+144.243814812" watchObservedRunningTime="2026-07-23 18:01:15.871293194 +0000 UTC m=+144.245956374" Jul 23 18:01:16.047106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:16.047073 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:01:16.841077 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:16.841030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4zgm" event={"ID":"d777d537-fec3-4930-8c4f-697074fdc4b6","Type":"ContainerStarted","Data":"fabb23eb8a1b04e34394bb46ba0ba0d8e93657ef8ba41ad75e005f4199e4ffb1"} Jul 23 18:01:16.841630 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:16.841104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4zgm" event={"ID":"d777d537-fec3-4930-8c4f-697074fdc4b6","Type":"ContainerStarted","Data":"650745113cc1da970abbb56da419caa03043962c71814401e14a85a80851f4ab"} Jul 23 18:01:16.868829 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:16.868732 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h4zgm" podStartSLOduration=10.339177718 podStartE2EDuration="16.868713088s" totalImagesPullingTime="6.389509012s" totalInitContainerRuntime="140.026358ms" isStatefulPod=true podCreationTimestamp="2026-07-23 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:16.867398609 +0000 UTC m=+145.242061789" watchObservedRunningTime="2026-07-23 18:01:16.868713088 +0000 UTC m=+145.243376269" Jul 23 18:01:19.854564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:19.854532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} Jul 23 18:01:19.861646 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:19.857139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f"} Jul 23 18:01:20.867117 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:20.867016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88"} Jul 23 18:01:20.867117 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:20.867111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2"} Jul 23 18:01:20.867755 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:20.867157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c"} Jul 23 18:01:20.867755 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:20.867201 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713"} Jul 23 18:01:20.869839 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:20.869810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} Jul 23 18:01:22.880702 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.880646 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} Jul 23 18:01:22.881216 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.880731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} Jul 23 18:01:22.881216 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.880772 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} Jul 23 18:01:22.881216 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.880810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerStarted","Data":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} Jul 23 18:01:22.883944 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.883917 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerStarted","Data":"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f"} Jul 23 18:01:22.916347 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.916296 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.588953958 podStartE2EDuration="16.916279422s" totalImagesPullingTime="12.29638028s" totalInitContainerRuntime="30.945184ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:22.909573856 +0000 UTC m=+151.284237035" watchObservedRunningTime="2026-07-23 18:01:22.916279422 +0000 UTC m=+151.290942602" Jul 23 18:01:22.942730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:22.942650 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.904007898 podStartE2EDuration="22.942636462s" totalImagesPullingTime="12.009753255s" totalInitContainerRuntime="28.875309ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:22.940153881 +0000 UTC m=+151.314817099" watchObservedRunningTime="2026-07-23 18:01:22.942636462 +0000 UTC m=+151.317299640" Jul 23 18:01:23.576540 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:23.576514 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bvbtd_ef4c5948-06e9-4626-8545-e242dde01132/dns-node-resolver/0.log" Jul 23 18:01:24.376098 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:24.376060 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnj5r_0437e524-41b9-4b65-a9aa-6f24c560e36d/serve-healthcheck-canary/0.log" Jul 23 18:01:25.009320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:25.009284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:25.009493 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:25.009344 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:26.533208 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:26.533168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:01:30.095262 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:01:30.095210 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-j7q2q" podUID="f2e796a2-1a03-48bf-85b6-e8a1f549b5fd" Jul 23 18:01:30.440285 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.440236 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b99b9575-zxw9g" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerName="console" containerID="cri-o://d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038" gracePeriod=15 Jul 23 18:01:30.712878 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.712804 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:01:30.772291 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772249 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rljx\" (UniqueName: \"kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772451 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772346 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772451 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772451 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772442 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772469 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772510 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert\") pod \"14e17f35-2f9c-4891-b603-7920937d8bbf\" (UID: \"14e17f35-2f9c-4891-b603-7920937d8bbf\") " Jul 23 18:01:30.772885 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772853 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config" (OuterVolumeSpecName: "console-config") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:30.772885 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca" (OuterVolumeSpecName: "service-ca") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:30.773038 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.772857 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:30.774578 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.774547 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:30.774662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.774585 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:30.774662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.774594 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx" (OuterVolumeSpecName: "kube-api-access-6rljx") pod "14e17f35-2f9c-4891-b603-7920937d8bbf" (UID: "14e17f35-2f9c-4891-b603-7920937d8bbf"). InnerVolumeSpecName "kube-api-access-6rljx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:01:30.873960 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873926 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-console-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.873960 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873955 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.873960 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873963 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e17f35-2f9c-4891-b603-7920937d8bbf-oauth-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.874207 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873973 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.874207 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873982 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rljx\" (UniqueName: \"kubernetes.io/projected/14e17f35-2f9c-4891-b603-7920937d8bbf-kube-api-access-6rljx\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.874207 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.873990 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e17f35-2f9c-4891-b603-7920937d8bbf-console-oauth-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:30.918464 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918430 2568 generic.go:356] "Generic (PLEG): container finished" podID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerID="d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038" exitCode=0 Jul 23 18:01:30.918659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918479 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99b9575-zxw9g" Jul 23 18:01:30.918659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99b9575-zxw9g" event={"ID":"14e17f35-2f9c-4891-b603-7920937d8bbf","Type":"ContainerDied","Data":"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038"} Jul 23 18:01:30.918659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918586 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99b9575-zxw9g" event={"ID":"14e17f35-2f9c-4891-b603-7920937d8bbf","Type":"ContainerDied","Data":"692df2f2d226b696a9be7ea116987d281ea55827a3f39071fbc56746dd881b16"} Jul 23 18:01:30.918659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918630 2568 scope.go:117] "RemoveContainer" containerID="d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038" Jul 23 18:01:30.918866 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.918748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j7q2q" Jul 23 18:01:30.928552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.928530 2568 scope.go:117] "RemoveContainer" containerID="d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038" Jul 23 18:01:30.928880 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:01:30.928860 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038\": container with ID starting with d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038 not found: ID does not exist" containerID="d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038" Jul 23 18:01:30.928971 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.928890 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038"} err="failed to get container status \"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038\": rpc error: code = NotFound desc = could not find container \"d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038\": container with ID starting with d858caf8e13643fc3dfbd88c8033e316c87089835ccb2bd97c11f8a574207038 not found: ID does not exist" Jul 23 18:01:30.951430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.947107 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:01:30.952169 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:30.952148 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b99b9575-zxw9g"] Jul 23 18:01:32.152079 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:32.152043 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" path="/var/lib/kubelet/pods/14e17f35-2f9c-4891-b603-7920937d8bbf/volumes" Jul 23 18:01:35.012100 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.012061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 18:01:35.014565 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.014531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2e796a2-1a03-48bf-85b6-e8a1f549b5fd-metrics-tls\") pod \"dns-default-j7q2q\" (UID: \"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd\") " pod="openshift-dns/dns-default-j7q2q" Jul 23 18:01:35.122429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.122401 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb24m\"" Jul 23 18:01:35.130406 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.130374 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j7q2q" Jul 23 18:01:35.322700 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.320927 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j7q2q"] Jul 23 18:01:35.336261 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:01:35.336195 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e796a2_1a03_48bf_85b6_e8a1f549b5fd.slice/crio-97401ae9ee4057f7cc6c249ec2158adb1ee109e84b30922e70ee4cc8f4c5962c WatchSource:0}: Error finding container 97401ae9ee4057f7cc6c249ec2158adb1ee109e84b30922e70ee4cc8f4c5962c: Status 404 returned error can't find the container with id 97401ae9ee4057f7cc6c249ec2158adb1ee109e84b30922e70ee4cc8f4c5962c Jul 23 18:01:35.953737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:35.953670 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j7q2q" event={"ID":"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd","Type":"ContainerStarted","Data":"97401ae9ee4057f7cc6c249ec2158adb1ee109e84b30922e70ee4cc8f4c5962c"} Jul 23 18:01:38.966372 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:38.966332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j7q2q" event={"ID":"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd","Type":"ContainerStarted","Data":"2a556838c27dc16d70af015884999909d2d77579b4f5cdc31f20edfd8b192b47"} Jul 23 18:01:38.966902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:38.966392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j7q2q" event={"ID":"f2e796a2-1a03-48bf-85b6-e8a1f549b5fd","Type":"ContainerStarted","Data":"b90e7c3f3b1a5757cc9a01b5790d9225ead9b305644d93ce5bbe4f7740ccf54e"} Jul 23 18:01:38.966902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:38.966434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j7q2q" Jul 23 18:01:38.988245 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:38.988199 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j7q2q" podStartSLOduration=129.342914076 podStartE2EDuration="2m11.988185195s" totalImagesPullingTime="2.645271119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:38.985803041 +0000 UTC m=+167.360466220" watchObservedRunningTime="2026-07-23 18:01:38.988185195 +0000 UTC m=+167.362848374" Jul 23 18:01:41.071920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.071878 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75766b8f9c-xl98s" podUID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" containerName="console" containerID="cri-o://7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f" gracePeriod=15 Jul 23 18:01:41.348103 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.348040 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:41.474349 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474539 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474381 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474609 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474546 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474609 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474586 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2hkq\" (UniqueName: \"kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474651 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474711 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca\") pod \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\" (UID: \"55e20eab-9d6b-4494-a7f5-0902c3452a7f\") " Jul 23 18:01:41.474949 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474915 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config" (OuterVolumeSpecName: "console-config") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:41.475019 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.474977 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:41.475154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.475125 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:41.475394 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.475365 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:41.476620 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.476594 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:41.476718 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.476624 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:41.476786 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.476761 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq" (OuterVolumeSpecName: "kube-api-access-c2hkq") pod "55e20eab-9d6b-4494-a7f5-0902c3452a7f" (UID: "55e20eab-9d6b-4494-a7f5-0902c3452a7f"). InnerVolumeSpecName "kube-api-access-c2hkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:01:41.575817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575777 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.575817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575811 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.575817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575822 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-oauth-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.575817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575831 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hkq\" (UniqueName: \"kubernetes.io/projected/55e20eab-9d6b-4494-a7f5-0902c3452a7f-kube-api-access-c2hkq\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.576082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575840 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.576082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575849 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55e20eab-9d6b-4494-a7f5-0902c3452a7f-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.576082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.575857 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55e20eab-9d6b-4494-a7f5-0902c3452a7f-console-oauth-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:01:41.979171 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.979130 2568 generic.go:356] "Generic (PLEG): container finished" podID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" containerID="7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f" exitCode=0 Jul 23 18:01:41.979362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.979200 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b8f9c-xl98s" Jul 23 18:01:41.979362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.979215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b8f9c-xl98s" event={"ID":"55e20eab-9d6b-4494-a7f5-0902c3452a7f","Type":"ContainerDied","Data":"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f"} Jul 23 18:01:41.979362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.979290 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b8f9c-xl98s" event={"ID":"55e20eab-9d6b-4494-a7f5-0902c3452a7f","Type":"ContainerDied","Data":"d9e634c74c90e7e5e4b74e574acc814bc18340d370723494b5c79397e7c4aaa5"} Jul 23 18:01:41.979362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.979312 2568 scope.go:117] "RemoveContainer" containerID="7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f" Jul 23 18:01:41.987905 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.987887 2568 scope.go:117] "RemoveContainer" containerID="7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f" Jul 23 18:01:41.988172 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:01:41.988153 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f\": container with ID starting with 7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f not found: ID does not exist" containerID="7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f" Jul 23 18:01:41.988219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:41.988180 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f"} err="failed to get container status \"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f\": rpc error: code = NotFound desc = could not find container \"7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f\": container with ID starting with 7326164815d369b90b8dc15990037aa85566cab5149225a3683fd350a645be0f not found: ID does not exist" Jul 23 18:01:42.002555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:42.002530 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:01:42.007621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:42.007599 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75766b8f9c-xl98s"] Jul 23 18:01:42.152974 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:42.152933 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" path="/var/lib/kubelet/pods/55e20eab-9d6b-4494-a7f5-0902c3452a7f/volumes" Jul 23 18:01:45.015319 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:45.015248 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:45.019129 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:45.019105 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fb598c4fd-z5xht" Jul 23 18:01:48.972628 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:01:48.972596 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j7q2q" Jul 23 18:02:06.533567 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:06.533523 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:06.552631 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:06.552608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:07.086966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:07.086940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:20.224621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.224583 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:20.225143 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.224876 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="alertmanager" containerID="cri-o://ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f" gracePeriod=120 Jul 23 18:02:20.225143 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.224938 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-metric" containerID="cri-o://df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88" gracePeriod=120 Jul 23 18:02:20.225595 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.225333 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-web" containerID="cri-o://ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c" gracePeriod=120 Jul 23 18:02:20.225595 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.225551 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy" containerID="cri-o://30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2" gracePeriod=120 Jul 23 18:02:20.225766 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.225719 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="config-reloader" containerID="cri-o://b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713" gracePeriod=120 Jul 23 18:02:20.225980 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:20.225867 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="prom-label-proxy" containerID="cri-o://cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f" gracePeriod=120 Jul 23 18:02:21.126430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126390 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f" exitCode=0 Jul 23 18:02:21.126430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126415 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88" exitCode=0 Jul 23 18:02:21.126430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126421 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2" exitCode=0 Jul 23 18:02:21.126430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126426 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713" exitCode=0 Jul 23 18:02:21.126430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126431 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f" exitCode=0 Jul 23 18:02:21.126848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126457 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f"} Jul 23 18:02:21.126848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88"} Jul 23 18:02:21.126848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2"} Jul 23 18:02:21.126848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713"} Jul 23 18:02:21.126848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.126741 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f"} Jul 23 18:02:21.471158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.471134 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:21.504995 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.504970 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlqjf\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505005 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505042 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505075 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505268 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505110 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505268 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505138 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505268 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505169 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505268 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505201 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505272 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505301 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505336 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505374 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca\") pod \"ed27d456-7901-49d7-8ac7-60f7a1be4771\" (UID: \"ed27d456-7901-49d7-8ac7-60f7a1be4771\") " Jul 23 18:02:21.505642 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505433 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:21.505719 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505675 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-main-db\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.505788 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505764 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:21.505882 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.505856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:21.508147 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508054 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out" (OuterVolumeSpecName: "config-out") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:21.508147 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508128 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf" (OuterVolumeSpecName: "kube-api-access-dlqjf") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "kube-api-access-dlqjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:02:21.508355 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508225 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.508355 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508288 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.508355 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508346 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume" (OuterVolumeSpecName: "config-volume") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.508703 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508659 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.508969 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.508946 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:02:21.509617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.509597 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.512901 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.512872 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.518631 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.518611 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config" (OuterVolumeSpecName: "web-config") pod "ed27d456-7901-49d7-8ac7-60f7a1be4771" (UID: "ed27d456-7901-49d7-8ac7-60f7a1be4771"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:21.606576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606545 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-cluster-tls-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606573 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-web-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606585 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606594 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-metrics-client-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606604 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqjf\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-kube-api-access-dlqjf\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606612 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-out\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606621 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606629 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed27d456-7901-49d7-8ac7-60f7a1be4771-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606637 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606658 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-config-volume\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606665 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed27d456-7901-49d7-8ac7-60f7a1be4771-secret-alertmanager-main-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:21.606828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:21.606672 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed27d456-7901-49d7-8ac7-60f7a1be4771-tls-assets\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:22.133373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.133338 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerID="ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c" exitCode=0 Jul 23 18:02:22.133527 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.133408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c"} Jul 23 18:02:22.133527 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.133418 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.133527 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.133473 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed27d456-7901-49d7-8ac7-60f7a1be4771","Type":"ContainerDied","Data":"bad9368881807a4a32366dc2a00293130d8c2780c76c5f2e7a23b774fa5faaa1"} Jul 23 18:02:22.133527 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.133505 2568 scope.go:117] "RemoveContainer" containerID="cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f" Jul 23 18:02:22.140912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.140891 2568 scope.go:117] "RemoveContainer" containerID="df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88" Jul 23 18:02:22.148039 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.148019 2568 scope.go:117] "RemoveContainer" containerID="30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2" Jul 23 18:02:22.155040 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.155022 2568 scope.go:117] "RemoveContainer" containerID="ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c" Jul 23 18:02:22.161359 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.161343 2568 scope.go:117] "RemoveContainer" containerID="b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713" Jul 23 18:02:22.168247 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.168183 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:22.169569 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.169550 2568 scope.go:117] "RemoveContainer" containerID="ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f" Jul 23 18:02:22.175462 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.175441 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:22.179099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.179079 2568 scope.go:117] "RemoveContainer" containerID="f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea" Jul 23 18:02:22.186794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.186774 2568 scope.go:117] "RemoveContainer" containerID="cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f" Jul 23 18:02:22.187029 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.187009 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f\": container with ID starting with cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f not found: ID does not exist" containerID="cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f" Jul 23 18:02:22.187078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187038 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f"} err="failed to get container status \"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f\": rpc error: code = NotFound desc = could not find container \"cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f\": container with ID starting with cea09194f0bd79cb413e389d7633252beb0987b8f8e6e66697d76f8167d3674f not found: ID does not exist" Jul 23 18:02:22.187078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187056 2568 scope.go:117] "RemoveContainer" containerID="df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88" Jul 23 18:02:22.187281 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.187263 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88\": container with ID starting with df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88 not found: ID does not exist" containerID="df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88" Jul 23 18:02:22.187324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187289 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88"} err="failed to get container status \"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88\": rpc error: code = NotFound desc = could not find container \"df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88\": container with ID starting with df7d88419332f04a0da13098374d151bd82acacef4f621c26a323543817ead88 not found: ID does not exist" Jul 23 18:02:22.187324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187305 2568 scope.go:117] "RemoveContainer" containerID="30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2" Jul 23 18:02:22.187521 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.187503 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2\": container with ID starting with 30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2 not found: ID does not exist" containerID="30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2" Jul 23 18:02:22.187562 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187527 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2"} err="failed to get container status \"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2\": rpc error: code = NotFound desc = could not find container \"30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2\": container with ID starting with 30635b3de485385a601b657af66cdf06818855a90ef36de97c6c32f6c87291f2 not found: ID does not exist" Jul 23 18:02:22.187562 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187549 2568 scope.go:117] "RemoveContainer" containerID="ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c" Jul 23 18:02:22.187768 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.187752 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c\": container with ID starting with ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c not found: ID does not exist" containerID="ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c" Jul 23 18:02:22.187819 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187775 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c"} err="failed to get container status \"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c\": rpc error: code = NotFound desc = could not find container \"ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c\": container with ID starting with ba6638d702143b5bd7add5d7a407d6dc943eab179fd7d26337be29a4f54fc00c not found: ID does not exist" Jul 23 18:02:22.187819 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.187790 2568 scope.go:117] "RemoveContainer" containerID="b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713" Jul 23 18:02:22.188007 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.187990 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713\": container with ID starting with b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713 not found: ID does not exist" containerID="b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713" Jul 23 18:02:22.188046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.188011 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713"} err="failed to get container status \"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713\": rpc error: code = NotFound desc = could not find container \"b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713\": container with ID starting with b768c2f778c8e5ed25dd8fbc9db6dc878f79b69aa40d03078eab3ff9a7b8b713 not found: ID does not exist" Jul 23 18:02:22.188046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.188026 2568 scope.go:117] "RemoveContainer" containerID="ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f" Jul 23 18:02:22.188225 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.188210 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f\": container with ID starting with ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f not found: ID does not exist" containerID="ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f" Jul 23 18:02:22.188265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.188228 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f"} err="failed to get container status \"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f\": rpc error: code = NotFound desc = could not find container \"ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f\": container with ID starting with ae21b4f6661494c4566a7671e71d230f9d6ca383d165811f377aeafd1530c18f not found: ID does not exist" Jul 23 18:02:22.188265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.188242 2568 scope.go:117] "RemoveContainer" containerID="f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea" Jul 23 18:02:22.188492 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:22.188477 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea\": container with ID starting with f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea not found: ID does not exist" containerID="f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea" Jul 23 18:02:22.188544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.188495 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea"} err="failed to get container status \"f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea\": rpc error: code = NotFound desc = could not find container \"f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea\": container with ID starting with f9b59b66bf7c35f49e1a6b1648994ac2f75e30c7a182030a81c40778aa2af1ea not found: ID does not exist" Jul 23 18:02:22.216861 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.216837 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:22.217089 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217078 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-metric" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217090 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-metric" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217098 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" containerName="console" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217103 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" containerName="console" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217110 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="init-config-reloader" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217116 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="init-config-reloader" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217122 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerName="console" Jul 23 18:02:22.217134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217127 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerName="console" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217138 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="config-reloader" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217143 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="config-reloader" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217148 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-web" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217154 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-web" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217162 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="alertmanager" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217167 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="alertmanager" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217180 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="prom-label-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217185 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="prom-label-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217193 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217198 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217233 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-web" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217241 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e20eab-9d6b-4494-a7f5-0902c3452a7f" containerName="console" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217247 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="config-reloader" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217252 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="alertmanager" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217258 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217263 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="prom-label-proxy" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217268 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e17f35-2f9c-4891-b603-7920937d8bbf" containerName="console" Jul 23 18:02:22.217339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.217273 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" containerName="kube-rbac-proxy-metric" Jul 23 18:02:22.220558 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.220543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.223134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.223118 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 18:02:22.223560 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.223545 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 18:02:22.223908 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.223891 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 18:02:22.223991 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.223916 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pmb2r\"" Jul 23 18:02:22.224048 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.223991 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 18:02:22.224048 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.224014 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 18:02:22.224552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.224531 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 18:02:22.224645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.224553 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 18:02:22.224645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.224601 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 18:02:22.229537 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.229519 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 18:02:22.252192 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.252165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:22.311519 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311519 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-web-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v522q\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-kube-api-access-v522q\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-out\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-volume\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311896 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.311957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.312157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.311959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.412988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-web-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v522q\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-kube-api-access-v522q\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-out\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-volume\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.413626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.413373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.414062 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.414036 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.414600 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.414552 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.414910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.414886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.416154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.416124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-web-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.416248 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.416124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.416595 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.416577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.417155 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.417133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.417265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.417156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-volume\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.417265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.417229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.418033 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.418011 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.418130 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.418064 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-config-out\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.418817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.418798 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.423973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.423952 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v522q\" (UniqueName: \"kubernetes.io/projected/a125d2fd-5c7b-44a9-bd64-53c2ec80a195-kube-api-access-v522q\") pod \"alertmanager-main-0\" (UID: \"a125d2fd-5c7b-44a9-bd64-53c2ec80a195\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.530247 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.530212 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 18:02:22.672177 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:22.672094 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 18:02:22.674263 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:02:22.674236 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda125d2fd_5c7b_44a9_bd64_53c2ec80a195.slice/crio-989dbaef92890d00eb2383fa1929cf893504ebe0c0d0c1b1201c4892a83076a3 WatchSource:0}: Error finding container 989dbaef92890d00eb2383fa1929cf893504ebe0c0d0c1b1201c4892a83076a3: Status 404 returned error can't find the container with id 989dbaef92890d00eb2383fa1929cf893504ebe0c0d0c1b1201c4892a83076a3 Jul 23 18:02:23.138798 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:23.138764 2568 generic.go:356] "Generic (PLEG): container finished" podID="a125d2fd-5c7b-44a9-bd64-53c2ec80a195" containerID="8bbc70078060d798024ad8eece9149cf4b552c0c5bffa63a1058d3a40ed8cfa9" exitCode=0 Jul 23 18:02:23.138968 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:23.138856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerDied","Data":"8bbc70078060d798024ad8eece9149cf4b552c0c5bffa63a1058d3a40ed8cfa9"} Jul 23 18:02:23.138968 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:23.138943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"989dbaef92890d00eb2383fa1929cf893504ebe0c0d0c1b1201c4892a83076a3"} Jul 23 18:02:24.152557 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.152525 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed27d456-7901-49d7-8ac7-60f7a1be4771" path="/var/lib/kubelet/pods/ed27d456-7901-49d7-8ac7-60f7a1be4771/volumes" Jul 23 18:02:24.153029 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"4c8da5651dad5d8fa48d445898344c6f759c73dcdb7d05af937cb5e1a71eb88b"} Jul 23 18:02:24.153078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"f7b90b87df136221fb7d8a1fb4bd1a88e63a6d3d5cf6fc6b12837947c0373180"} Jul 23 18:02:24.153115 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"847baba6785f1176c121daf97084f776958cee09deed6875e7e73a6fbadbc21c"} Jul 23 18:02:24.153154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153118 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"a1ef8ebd9eb30288313cefa24247cb92f8350996a9f7eba0aadc213eaf74a566"} Jul 23 18:02:24.153154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"2855e6878f8a117e9fa7e2576d3f39484bd8aca7c5db115635da607852cc4528"} Jul 23 18:02:24.153219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.153165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a125d2fd-5c7b-44a9-bd64-53c2ec80a195","Type":"ContainerStarted","Data":"25050a38f59a2286c2df9fa5aa9647a0274b3d0fe2ceff4af10042ad84276c6c"} Jul 23 18:02:24.205279 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.205204 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.190928975 podStartE2EDuration="2.205188917s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.259942ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:02:24.203813713 +0000 UTC m=+212.578476893" watchObservedRunningTime="2026-07-23 18:02:24.205188917 +0000 UTC m=+212.579852095" Jul 23 18:02:24.619457 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619366 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:24.619743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619705 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="prometheus" containerID="cri-o://bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" gracePeriod=600 Jul 23 18:02:24.619912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619735 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy" containerID="cri-o://6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" gracePeriod=600 Jul 23 18:02:24.619912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619767 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-web" containerID="cri-o://c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" gracePeriod=600 Jul 23 18:02:24.619912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619791 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="config-reloader" containerID="cri-o://251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" gracePeriod=600 Jul 23 18:02:24.619912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619768 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" gracePeriod=600 Jul 23 18:02:24.619912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.619764 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="thanos-sidecar" containerID="cri-o://fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" gracePeriod=600 Jul 23 18:02:24.863181 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.863153 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:24.939290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939260 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939293 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939315 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939357 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939376 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939402 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939429 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939544 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939453 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.939852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939660 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.939949 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.939927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940037 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940021 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940093 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940057 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940098 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940126 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940237 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940155 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940237 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940153 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.940237 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940198 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940237 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940230 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940269 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940295 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw74d\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940327 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940351 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls\") pod \"2a854d20-ddbf-457d-8016-c3379fffce9a\" (UID: \"2a854d20-ddbf-457d-8016-c3379fffce9a\") " Jul 23 18:02:24.940627 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940609 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:24.940694 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940633 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-metrics-client-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:24.941020 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.940973 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.941320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.941293 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.942573 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.942544 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.942573 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.942548 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.942800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.942658 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:24.942800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.942766 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.943124 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.943098 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:02:24.943699 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.943654 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out" (OuterVolumeSpecName: "config-out") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:24.944027 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.943998 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.944169 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.944145 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:02:24.944381 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.944351 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.944543 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.944524 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.945195 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.945168 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.945281 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.945191 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d" (OuterVolumeSpecName: "kube-api-access-zw74d") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "kube-api-access-zw74d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:02:24.945486 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.945461 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.945486 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.945476 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config" (OuterVolumeSpecName: "config") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.945610 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.945580 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:24.954650 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:24.954622 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config" (OuterVolumeSpecName: "web-config") pod "2a854d20-ddbf-457d-8016-c3379fffce9a" (UID: "2a854d20-ddbf-457d-8016-c3379fffce9a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:02:25.041325 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041289 2568 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041325 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041319 2568 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-metrics-client-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041325 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041331 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-db\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041340 2568 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-kube-rbac-proxy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041349 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041359 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-web-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041368 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041376 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw74d\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-kube-api-access-zw74d\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041384 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041392 2568 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-grpc-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041400 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041408 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a854d20-ddbf-457d-8016-c3379fffce9a-config-out\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041417 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041425 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041433 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a854d20-ddbf-457d-8016-c3379fffce9a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041441 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a854d20-ddbf-457d-8016-c3379fffce9a-tls-assets\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041450 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.041542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.041458 2568 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a854d20-ddbf-457d-8016-c3379fffce9a-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:02:25.154750 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154715 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" exitCode=0 Jul 23 18:02:25.154750 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154741 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" exitCode=0 Jul 23 18:02:25.154750 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154747 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" exitCode=0 Jul 23 18:02:25.154750 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154754 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" exitCode=0 Jul 23 18:02:25.154750 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154760 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" exitCode=0 Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154768 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" exitCode=0 Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154792 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154855 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154889 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.154971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.155000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} Jul 23 18:02:25.155342 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.155026 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a854d20-ddbf-457d-8016-c3379fffce9a","Type":"ContainerDied","Data":"ead16eb5db2b03fd256340c7150a39301208dd25179e9501bf3b90ffe63d4f82"} Jul 23 18:02:25.163427 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.163228 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.170114 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.170093 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.176470 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.176452 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.182578 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.182556 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.185266 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.185247 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:25.189651 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.189636 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.190224 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.190202 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:25.196536 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.196520 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.202824 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.202808 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.203078 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.203061 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.203123 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203085 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.203123 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203103 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.203340 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.203324 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.203421 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203342 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.203421 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203355 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.203541 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.203528 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.203576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203544 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.203576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203557 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.203796 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.203776 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.203851 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203803 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.203851 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.203825 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.204074 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.204057 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.204109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204079 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.204109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204091 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.204312 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.204298 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.204354 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204313 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.204354 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204325 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.204561 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:02:25.204545 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.204605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204564 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.204605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204581 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.204806 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204786 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.204806 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204803 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.205010 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.204986 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.205053 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205012 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.205250 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205221 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.205250 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205242 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.205459 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205444 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.205517 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205459 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.205667 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205640 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.205667 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205660 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.205916 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205900 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.205961 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.205918 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.206156 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206136 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.206196 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206156 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.206365 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206348 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.206418 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206364 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.206560 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206546 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.206613 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206560 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.206811 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206792 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.206862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.206811 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.207027 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207009 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.207078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207028 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.207240 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207226 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.207240 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207240 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.207455 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207428 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.207455 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207447 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.207660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207639 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.207660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207660 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.207902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207884 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.207942 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.207903 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.208110 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208091 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.208150 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208111 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.208361 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208340 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.208438 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208362 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.208621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208602 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.208692 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208622 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.208950 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208932 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.208950 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.208949 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.209165 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209143 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.209210 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209165 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.209356 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209338 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.209403 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209357 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.209563 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209546 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.209607 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209564 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.209805 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209785 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.209805 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.209805 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.210035 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210018 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.210081 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210036 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.210232 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210213 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.210288 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210232 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.210422 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210404 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.210464 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210423 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.210630 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210611 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.210708 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210633 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.210920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210904 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.210972 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.210921 2568 scope.go:117] "RemoveContainer" containerID="f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843" Jul 23 18:02:25.211127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211109 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843"} err="failed to get container status \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": rpc error: code = NotFound desc = could not find container \"f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843\": container with ID starting with f18c6081bd5287be1d5277626667ad82fc890aeb621aa340675bc544b27ee843 not found: ID does not exist" Jul 23 18:02:25.211166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211133 2568 scope.go:117] "RemoveContainer" containerID="6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495" Jul 23 18:02:25.211339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211324 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495"} err="failed to get container status \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": rpc error: code = NotFound desc = could not find container \"6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495\": container with ID starting with 6d08fcefaa5c8f885d3219dfcc6c997dc0858cfc9c0b2be16368a89e1c227495 not found: ID does not exist" Jul 23 18:02:25.211386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211339 2568 scope.go:117] "RemoveContainer" containerID="c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb" Jul 23 18:02:25.211527 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211512 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb"} err="failed to get container status \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": rpc error: code = NotFound desc = could not find container \"c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb\": container with ID starting with c0b6acb9f1a89ce45ac5463b29732ab32d843dd95a19696caca93cb4645e07bb not found: ID does not exist" Jul 23 18:02:25.211566 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211527 2568 scope.go:117] "RemoveContainer" containerID="fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77" Jul 23 18:02:25.211732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211712 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77"} err="failed to get container status \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": rpc error: code = NotFound desc = could not find container \"fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77\": container with ID starting with fe192353ae2275a362709b3cd91e50ff947d046876f35c211f0fddb445203b77 not found: ID does not exist" Jul 23 18:02:25.211788 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211733 2568 scope.go:117] "RemoveContainer" containerID="251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb" Jul 23 18:02:25.211977 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211960 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb"} err="failed to get container status \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": rpc error: code = NotFound desc = could not find container \"251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb\": container with ID starting with 251de5ebf417e3a01f5a661120d474729738c1b7b8068733a1a01a07274274bb not found: ID does not exist" Jul 23 18:02:25.212023 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.211978 2568 scope.go:117] "RemoveContainer" containerID="bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57" Jul 23 18:02:25.212201 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.212183 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57"} err="failed to get container status \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": rpc error: code = NotFound desc = could not find container \"bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57\": container with ID starting with bd18cbaaeb2d71f6adedbf2ad567b2b056803f17e458e23007c987b87f5d5e57 not found: ID does not exist" Jul 23 18:02:25.212243 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.212202 2568 scope.go:117] "RemoveContainer" containerID="eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96" Jul 23 18:02:25.212413 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.212395 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96"} err="failed to get container status \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": rpc error: code = NotFound desc = could not find container \"eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96\": container with ID starting with eae686dbfdb4d1a35a10ccd0adef4852750a7c68a9b244f140365620cb113d96 not found: ID does not exist" Jul 23 18:02:25.218168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218149 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:25.218400 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218389 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="config-reloader" Jul 23 18:02:25.218435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218402 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="config-reloader" Jul 23 18:02:25.218435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218412 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-web" Jul 23 18:02:25.218435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218418 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-web" Jul 23 18:02:25.218435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218426 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="init-config-reloader" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218445 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="init-config-reloader" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218452 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-thanos" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218457 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-thanos" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218468 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="prometheus" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218472 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="prometheus" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218479 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="thanos-sidecar" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218484 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="thanos-sidecar" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218495 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218501 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218540 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="prometheus" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218547 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-thanos" Jul 23 18:02:25.218554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218554 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="thanos-sidecar" Jul 23 18:02:25.218920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218560 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="config-reloader" Jul 23 18:02:25.218920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218565 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy-web" Jul 23 18:02:25.218920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.218569 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" containerName="kube-rbac-proxy" Jul 23 18:02:25.222132 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.222118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.224945 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.224927 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 18:02:25.225524 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 18:02:25.225524 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225513 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 18:02:25.225677 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225518 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 18:02:25.225677 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225647 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 18:02:25.225677 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225658 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 18:02:25.225852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.225806 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 18:02:25.226516 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.226499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 18:02:25.226606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.226591 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39qesgq6uev3p\"" Jul 23 18:02:25.226928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.226911 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ldvpn\"" Jul 23 18:02:25.227004 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.226932 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 18:02:25.227056 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.227015 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 18:02:25.227056 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.227018 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 18:02:25.227208 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.227190 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 18:02:25.228942 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.228928 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 18:02:25.233057 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.233036 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 18:02:25.245632 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.245613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:25.344542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344538 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344557 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344574 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68h7\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-kube-api-access-g68h7\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344759 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344893 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.344989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.345180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.344999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.345180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.345015 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.345180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.345052 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.445924 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.445827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.445924 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.445878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.445924 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.445895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446228 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446255 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446360 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68h7\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-kube-api-access-g68h7\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446419 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.446879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.446865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.447623 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.447341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.448958 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.448929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449218 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-web-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e09b363-b25f-42d0-b01b-615f2ceab43c-config-out\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.449976 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.450045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.450286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.450291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.450568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.450530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e09b363-b25f-42d0-b01b-615f2ceab43c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.451367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.451343 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-config\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.451654 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.451637 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.451801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.451785 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.452134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.452113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e09b363-b25f-42d0-b01b-615f2ceab43c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.457608 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.457592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68h7\" (UniqueName: \"kubernetes.io/projected/7e09b363-b25f-42d0-b01b-615f2ceab43c-kube-api-access-g68h7\") pod \"prometheus-k8s-0\" (UID: \"7e09b363-b25f-42d0-b01b-615f2ceab43c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.532531 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.532491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:25.677374 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:25.677343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 18:02:25.679639 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:02:25.679609 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e09b363_b25f_42d0_b01b_615f2ceab43c.slice/crio-f1112d99a4173947699ab3a2f69bcd0a9a80e79112573f557638a7e6f5be1c09 WatchSource:0}: Error finding container f1112d99a4173947699ab3a2f69bcd0a9a80e79112573f557638a7e6f5be1c09: Status 404 returned error can't find the container with id f1112d99a4173947699ab3a2f69bcd0a9a80e79112573f557638a7e6f5be1c09 Jul 23 18:02:26.152067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:26.152037 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a854d20-ddbf-457d-8016-c3379fffce9a" path="/var/lib/kubelet/pods/2a854d20-ddbf-457d-8016-c3379fffce9a/volumes" Jul 23 18:02:26.164964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:26.164931 2568 generic.go:356] "Generic (PLEG): container finished" podID="7e09b363-b25f-42d0-b01b-615f2ceab43c" containerID="6aeed254080969d6e570c1a62f18b59813037f1a3d7154e876293dd61da61f45" exitCode=0 Jul 23 18:02:26.165357 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:26.165017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerDied","Data":"6aeed254080969d6e570c1a62f18b59813037f1a3d7154e876293dd61da61f45"} Jul 23 18:02:26.165357 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:26.165098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"f1112d99a4173947699ab3a2f69bcd0a9a80e79112573f557638a7e6f5be1c09"} Jul 23 18:02:27.175791 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"d373c777e0fb90d41ac06a75c2a56e55e7c02d251f209f0d13191c24c320e17b"} Jul 23 18:02:27.176265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"fc2fcd9d1e059cd11738b28ea4991d2144ee8bd0d18899720c493e89e0bef619"} Jul 23 18:02:27.176265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"3618773064f979681d090f84c99e7cb578380b55537de4f64a74988e170f7e66"} Jul 23 18:02:27.176265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"528a2046247363c523b5a0aa7d8b5267237178bf059c5ae30fe1176168fafa71"} Jul 23 18:02:27.176265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"96e32d88f341b906e9f4f8d3ff6cd472d0e35bea9da0210ca1d85074c52f0487"} Jul 23 18:02:27.176265 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.175958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7e09b363-b25f-42d0-b01b-615f2ceab43c","Type":"ContainerStarted","Data":"82631545db0887239b5affe47b36ff887727e7d1dd2480e2c0f341f324301e0a"} Jul 23 18:02:27.212447 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:27.212393 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.193682846 podStartE2EDuration="2.212377972s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.695126ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:02:27.211101962 +0000 UTC m=+215.585765175" watchObservedRunningTime="2026-07-23 18:02:27.212377972 +0000 UTC m=+215.587041150" Jul 23 18:02:30.533480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.533439 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:02:30.844857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.844771 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:02:30.847338 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.847321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.861373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.861349 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:02:30.893640 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893614 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893710 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893899 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893805 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.893899 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.893828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2gt\" (UniqueName: \"kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.994959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.994921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.994959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.994959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.994977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2gt\" (UniqueName: \"kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995671 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995780 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995807 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.995951 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.995924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.996075 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.996057 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.997539 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.997515 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:30.997621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:30.997597 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:31.003605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:31.003582 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2gt\" (UniqueName: \"kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt\") pod \"console-b7bdf4c47-k25ws\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:31.157067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:31.157031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:31.299663 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:31.299632 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:02:32.195972 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:32.195937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7bdf4c47-k25ws" event={"ID":"4e5a4f30-b912-405b-8503-912f73979d1a","Type":"ContainerStarted","Data":"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a"} Jul 23 18:02:32.196350 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:32.195993 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7bdf4c47-k25ws" event={"ID":"4e5a4f30-b912-405b-8503-912f73979d1a","Type":"ContainerStarted","Data":"58e96a5af057cced285003e08da7a770630acc0099ecf8700cb725bc977ca5da"} Jul 23 18:02:32.217781 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:32.217728 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b7bdf4c47-k25ws" podStartSLOduration=2.217713532 podStartE2EDuration="2.217713532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:02:32.216171314 +0000 UTC m=+220.590834505" watchObservedRunningTime="2026-07-23 18:02:32.217713532 +0000 UTC m=+220.592376798" Jul 23 18:02:41.157570 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:41.157530 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:41.158084 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:41.157605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:41.162599 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:41.162574 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:41.234424 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:41.234394 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:02:41.284734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:02:41.284705 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:03:06.305392 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.305338 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7666b4b565-6chfh" podUID="5928ac32-439b-49c0-8004-212f8c32ff3d" containerName="console" containerID="cri-o://01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a" gracePeriod=15 Jul 23 18:03:06.550545 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.550516 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:03:06.595450 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595374 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595450 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595432 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595661 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595661 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595497 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595661 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595532 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595839 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595664 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5clk\" (UniqueName: \"kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.595839 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595731 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config\") pod \"5928ac32-439b-49c0-8004-212f8c32ff3d\" (UID: \"5928ac32-439b-49c0-8004-212f8c32ff3d\") " Jul 23 18:03:06.596008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595960 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:03:06.596008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595974 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:03:06.596008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.595967 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:03:06.596370 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.596271 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config" (OuterVolumeSpecName: "console-config") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:03:06.597648 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.597624 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:03:06.597648 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.597639 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:03:06.597948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.597644 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk" (OuterVolumeSpecName: "kube-api-access-l5clk") pod "5928ac32-439b-49c0-8004-212f8c32ff3d" (UID: "5928ac32-439b-49c0-8004-212f8c32ff3d"). InnerVolumeSpecName "kube-api-access-l5clk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:03:06.696496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696459 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5clk\" (UniqueName: \"kubernetes.io/projected/5928ac32-439b-49c0-8004-212f8c32ff3d-kube-api-access-l5clk\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696488 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-console-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696499 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696508 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696516 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696524 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5928ac32-439b-49c0-8004-212f8c32ff3d-console-oauth-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:06.696800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:06.696534 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5928ac32-439b-49c0-8004-212f8c32ff3d-oauth-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:03:07.329567 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.329533 2568 generic.go:356] "Generic (PLEG): container finished" podID="5928ac32-439b-49c0-8004-212f8c32ff3d" containerID="01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a" exitCode=0 Jul 23 18:03:07.329982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.329582 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7666b4b565-6chfh" Jul 23 18:03:07.329982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.329621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7666b4b565-6chfh" event={"ID":"5928ac32-439b-49c0-8004-212f8c32ff3d","Type":"ContainerDied","Data":"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a"} Jul 23 18:03:07.329982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.329672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7666b4b565-6chfh" event={"ID":"5928ac32-439b-49c0-8004-212f8c32ff3d","Type":"ContainerDied","Data":"127f7948fc5b85e6643d80bf5b17f66347e6540dc7111da01685624409c8bd0b"} Jul 23 18:03:07.329982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.329713 2568 scope.go:117] "RemoveContainer" containerID="01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a" Jul 23 18:03:07.338425 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.338406 2568 scope.go:117] "RemoveContainer" containerID="01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a" Jul 23 18:03:07.338665 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:03:07.338644 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a\": container with ID starting with 01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a not found: ID does not exist" containerID="01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a" Jul 23 18:03:07.338760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.338700 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a"} err="failed to get container status \"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a\": rpc error: code = NotFound desc = could not find container \"01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a\": container with ID starting with 01f7bcbac1066dabb4c71214bb940eab118f960d282adeaad40869ccca036f8a not found: ID does not exist" Jul 23 18:03:07.355626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.355599 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:03:07.366205 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:07.366182 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7666b4b565-6chfh"] Jul 23 18:03:08.151569 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:08.151534 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5928ac32-439b-49c0-8004-212f8c32ff3d" path="/var/lib/kubelet/pods/5928ac32-439b-49c0-8004-212f8c32ff3d/volumes" Jul 23 18:03:25.533112 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:25.533007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:03:25.548525 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:25.548498 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:03:26.418966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:26.418935 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 18:03:36.604963 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.604927 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:03:36.605347 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.605217 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5928ac32-439b-49c0-8004-212f8c32ff3d" containerName="console" Jul 23 18:03:36.605347 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.605229 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5928ac32-439b-49c0-8004-212f8c32ff3d" containerName="console" Jul 23 18:03:36.605347 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.605286 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5928ac32-439b-49c0-8004-212f8c32ff3d" containerName="console" Jul 23 18:03:36.609580 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.609557 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.623211 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.623186 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:03:36.765737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjzg\" (UniqueName: \"kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.765931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.765931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.765931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.765931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.766138 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.765945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.766138 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.766029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867013 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.866925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867013 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.866964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjzg\" (UniqueName: \"kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867713 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.867848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.867821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.868045 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.868022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.868085 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.868037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.869485 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.869458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.869579 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.869539 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.876491 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.876469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjzg\" (UniqueName: \"kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg\") pod \"console-664fccfcb7-2w7s6\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:36.918333 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:36.918296 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:37.057046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:37.057012 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:03:37.060033 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:03:37.060007 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ab2643_7a46_4201_833d_d60fd8ddf38d.slice/crio-892fb75bddabe404d2e3815558cf06a17f86e78bdfc0fb7d4b3916204744300b WatchSource:0}: Error finding container 892fb75bddabe404d2e3815558cf06a17f86e78bdfc0fb7d4b3916204744300b: Status 404 returned error can't find the container with id 892fb75bddabe404d2e3815558cf06a17f86e78bdfc0fb7d4b3916204744300b Jul 23 18:03:37.451040 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:37.451003 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664fccfcb7-2w7s6" event={"ID":"10ab2643-7a46-4201-833d-d60fd8ddf38d","Type":"ContainerStarted","Data":"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea"} Jul 23 18:03:37.451212 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:37.451060 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664fccfcb7-2w7s6" event={"ID":"10ab2643-7a46-4201-833d-d60fd8ddf38d","Type":"ContainerStarted","Data":"892fb75bddabe404d2e3815558cf06a17f86e78bdfc0fb7d4b3916204744300b"} Jul 23 18:03:37.473606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:37.473559 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-664fccfcb7-2w7s6" podStartSLOduration=1.473542527 podStartE2EDuration="1.473542527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:03:37.471343941 +0000 UTC m=+285.846007123" watchObservedRunningTime="2026-07-23 18:03:37.473542527 +0000 UTC m=+285.848205706" Jul 23 18:03:46.919209 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:46.919171 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:46.919628 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:46.919255 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:46.923897 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:46.923877 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:47.492150 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:47.492122 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:03:47.569597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:47.569566 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:03:52.055635 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:03:52.055607 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 18:04:12.589871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.589831 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b7bdf4c47-k25ws" podUID="4e5a4f30-b912-405b-8503-912f73979d1a" containerName="console" containerID="cri-o://77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a" gracePeriod=15 Jul 23 18:04:12.836932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.836908 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:04:12.994645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994610 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.994817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.994817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994720 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.994817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994746 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm2gt\" (UniqueName: \"kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.994817 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994784 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.994993 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.994954 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.995046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995005 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert\") pod \"4e5a4f30-b912-405b-8503-912f73979d1a\" (UID: \"4e5a4f30-b912-405b-8503-912f73979d1a\") " Jul 23 18:04:12.995105 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995080 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:04:12.995253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995216 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:04:12.995509 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995410 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:12.995509 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995431 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:12.995509 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995453 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:04:12.995509 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.995464 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config" (OuterVolumeSpecName: "console-config") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:04:12.996949 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.996922 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:04:12.996949 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.996937 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:04:12.997072 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:12.996958 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt" (OuterVolumeSpecName: "kube-api-access-mm2gt") pod "4e5a4f30-b912-405b-8503-912f73979d1a" (UID: "4e5a4f30-b912-405b-8503-912f73979d1a"). InnerVolumeSpecName "kube-api-access-mm2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:04:13.096659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.096619 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2gt\" (UniqueName: \"kubernetes.io/projected/4e5a4f30-b912-405b-8503-912f73979d1a-kube-api-access-mm2gt\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:13.096659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.096648 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:13.096659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.096658 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-console-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:13.096659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.096666 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a4f30-b912-405b-8503-912f73979d1a-oauth-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:13.096939 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.096676 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e5a4f30-b912-405b-8503-912f73979d1a-console-oauth-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:04:13.592269 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.592232 2568 generic.go:356] "Generic (PLEG): container finished" podID="4e5a4f30-b912-405b-8503-912f73979d1a" containerID="77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a" exitCode=0 Jul 23 18:04:13.592269 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.592272 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7bdf4c47-k25ws" Jul 23 18:04:13.592768 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.592318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7bdf4c47-k25ws" event={"ID":"4e5a4f30-b912-405b-8503-912f73979d1a","Type":"ContainerDied","Data":"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a"} Jul 23 18:04:13.592768 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.592369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7bdf4c47-k25ws" event={"ID":"4e5a4f30-b912-405b-8503-912f73979d1a","Type":"ContainerDied","Data":"58e96a5af057cced285003e08da7a770630acc0099ecf8700cb725bc977ca5da"} Jul 23 18:04:13.592768 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.592388 2568 scope.go:117] "RemoveContainer" containerID="77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a" Jul 23 18:04:13.602515 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.602492 2568 scope.go:117] "RemoveContainer" containerID="77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a" Jul 23 18:04:13.602886 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:04:13.602842 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a\": container with ID starting with 77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a not found: ID does not exist" containerID="77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a" Jul 23 18:04:13.602992 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.602884 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a"} err="failed to get container status \"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a\": rpc error: code = NotFound desc = could not find container \"77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a\": container with ID starting with 77a3f11dc7229eaebe2971317fbddcb116d72ae7cb969dd72054be228b089e2a not found: ID does not exist" Jul 23 18:04:13.619514 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.619479 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:04:13.622972 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:13.622945 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b7bdf4c47-k25ws"] Jul 23 18:04:14.152234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:04:14.152198 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5a4f30-b912-405b-8503-912f73979d1a" path="/var/lib/kubelet/pods/4e5a4f30-b912-405b-8503-912f73979d1a/volumes" Jul 23 18:06:12.551012 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.550981 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp"] Jul 23 18:06:12.551473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.551292 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5a4f30-b912-405b-8503-912f73979d1a" containerName="console" Jul 23 18:06:12.551473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.551303 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5a4f30-b912-405b-8503-912f73979d1a" containerName="console" Jul 23 18:06:12.551473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.551368 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5a4f30-b912-405b-8503-912f73979d1a" containerName="console" Jul 23 18:06:12.554152 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.554136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.556665 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.556643 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 18:06:12.556813 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.556744 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vn7x2\"" Jul 23 18:06:12.557531 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.557516 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 18:06:12.574954 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.574928 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp"] Jul 23 18:06:12.589981 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.589954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l94t\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-kube-api-access-8l94t\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.590087 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.590020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.690779 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.690736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l94t\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-kube-api-access-8l94t\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.690955 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.690813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.703047 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.703018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.703197 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.703178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l94t\" (UniqueName: \"kubernetes.io/projected/3df829e9-54fc-46c2-9ad2-ee58f38e663b-kube-api-access-8l94t\") pod \"cert-manager-cainjector-559cdd8c87-gnhdp\" (UID: \"3df829e9-54fc-46c2-9ad2-ee58f38e663b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.863249 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.863171 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" Jul 23 18:06:12.995751 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:12.995720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp"] Jul 23 18:06:12.998932 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:06:12.998905 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3df829e9_54fc_46c2_9ad2_ee58f38e663b.slice/crio-80b3779fe5b23e44a2773d32255d4ed0a179ef244d08dfc455e443ef57221340 WatchSource:0}: Error finding container 80b3779fe5b23e44a2773d32255d4ed0a179ef244d08dfc455e443ef57221340: Status 404 returned error can't find the container with id 80b3779fe5b23e44a2773d32255d4ed0a179ef244d08dfc455e443ef57221340 Jul 23 18:06:13.000807 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:13.000791 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:06:13.031225 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:13.031196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" event={"ID":"3df829e9-54fc-46c2-9ad2-ee58f38e663b","Type":"ContainerStarted","Data":"80b3779fe5b23e44a2773d32255d4ed0a179ef244d08dfc455e443ef57221340"} Jul 23 18:06:17.053673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:17.053632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" event={"ID":"3df829e9-54fc-46c2-9ad2-ee58f38e663b","Type":"ContainerStarted","Data":"9c6dad060893697017850df0a1a0f7655a1708f2dd3dd0707d0a11d9cb1aca29"} Jul 23 18:06:46.234704 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.234631 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnhdp" podStartSLOduration=30.940805049 podStartE2EDuration="34.234612893s" totalImagesPullingTime="3.293807844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:17.077413563 +0000 UTC m=+445.452076742" watchObservedRunningTime="2026-07-23 18:06:46.234612893 +0000 UTC m=+474.609276074" Jul 23 18:06:46.235180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.234949 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf"] Jul 23 18:06:46.238220 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.238203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.243087 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.243059 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 18:06:46.243380 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.243361 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 18:06:46.243519 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.243397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g6dq5\"" Jul 23 18:06:46.243939 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.243918 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 18:06:46.244043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.243992 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 18:06:46.254100 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.254079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf"] Jul 23 18:06:46.391880 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.391844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.392061 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.391897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjgf\" (UniqueName: \"kubernetes.io/projected/4c8926d5-8943-47e6-a83b-d18eb092d0db-kube-api-access-dxjgf\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.392061 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.391971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.392061 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.392001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.392187 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.392056 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjgf\" (UniqueName: \"kubernetes.io/projected/4c8926d5-8943-47e6-a83b-d18eb092d0db-kube-api-access-dxjgf\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.493837 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.493813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.495605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.495576 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8926d5-8943-47e6-a83b-d18eb092d0db-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.495835 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.495819 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.495883 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.495848 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c8926d5-8943-47e6-a83b-d18eb092d0db-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.504609 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.504590 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjgf\" (UniqueName: \"kubernetes.io/projected/4c8926d5-8943-47e6-a83b-d18eb092d0db-kube-api-access-dxjgf\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-fzzmf\" (UID: \"4c8926d5-8943-47e6-a83b-d18eb092d0db\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.547614 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.547585 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:46.684842 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:46.684810 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf"] Jul 23 18:06:46.688097 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:06:46.688070 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8926d5_8943_47e6_a83b_d18eb092d0db.slice/crio-415e29d377cb11ecccd0180074aeca050dd7c1e25f56af51374f208e53addffc WatchSource:0}: Error finding container 415e29d377cb11ecccd0180074aeca050dd7c1e25f56af51374f208e53addffc: Status 404 returned error can't find the container with id 415e29d377cb11ecccd0180074aeca050dd7c1e25f56af51374f208e53addffc Jul 23 18:06:47.171595 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:47.171561 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" event={"ID":"4c8926d5-8943-47e6-a83b-d18eb092d0db","Type":"ContainerStarted","Data":"415e29d377cb11ecccd0180074aeca050dd7c1e25f56af51374f208e53addffc"} Jul 23 18:06:51.195724 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:51.195664 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" event={"ID":"4c8926d5-8943-47e6-a83b-d18eb092d0db","Type":"ContainerStarted","Data":"eecee20b46b01073cded0f863256effbce4a103e4d87944b48de3068d0fbb30d"} Jul 23 18:06:52.201164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:52.201129 2568 generic.go:356] "Generic (PLEG): container finished" podID="4c8926d5-8943-47e6-a83b-d18eb092d0db" containerID="eecee20b46b01073cded0f863256effbce4a103e4d87944b48de3068d0fbb30d" exitCode=0 Jul 23 18:06:52.201595 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:52.201205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" event={"ID":"4c8926d5-8943-47e6-a83b-d18eb092d0db","Type":"ContainerDied","Data":"eecee20b46b01073cded0f863256effbce4a103e4d87944b48de3068d0fbb30d"} Jul 23 18:06:53.206770 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:53.206720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" event={"ID":"4c8926d5-8943-47e6-a83b-d18eb092d0db","Type":"ContainerStarted","Data":"0686388589d93aa3c011c33a6263d2e9684f6ce31a1a00cb69ad3fe764ea7a11"} Jul 23 18:06:53.207171 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:53.206869 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:06:53.238247 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:06:53.238202 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" podStartSLOduration=2.35407498 podStartE2EDuration="7.238187946s" totalImagesPullingTime="3.818058633s" totalInitContainerRuntime="1.066054333s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:53.237699427 +0000 UTC m=+481.612362604" watchObservedRunningTime="2026-07-23 18:06:53.238187946 +0000 UTC m=+481.612851124" Jul 23 18:07:04.213353 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:04.213312 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-fzzmf" Jul 23 18:07:15.743456 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.743423 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p"] Jul 23 18:07:15.745833 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.745817 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.750488 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.750460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 18:07:15.750626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.750516 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 18:07:15.750626 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.750527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 18:07:15.750928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.750910 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-77tr6\"" Jul 23 18:07:15.751037 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.750939 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:07:15.751146 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.751131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 18:07:15.762736 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.762711 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p"] Jul 23 18:07:15.838443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.838409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-manager-config\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.838443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.838441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.838722 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.838586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-metrics-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.838722 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.838633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zmx\" (UniqueName: \"kubernetes.io/projected/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-kube-api-access-99zmx\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.940122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.940093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-metrics-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.940275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.940137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zmx\" (UniqueName: \"kubernetes.io/projected/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-kube-api-access-99zmx\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.940275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.940161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-manager-config\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.940275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.940186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.941082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.941048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-manager-config\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.942834 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.942811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-metrics-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.942941 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.942847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-cert\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.960294 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.960267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zmx\" (UniqueName: \"kubernetes.io/projected/5245db9c-8eb0-49d6-8e24-35b8d9b79c12-kube-api-access-99zmx\") pod \"lws-controller-manager-7ff5b44f4d-2rp6p\" (UID: \"5245db9c-8eb0-49d6-8e24-35b8d9b79c12\") " pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:15.976516 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.976489 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l"] Jul 23 18:07:15.978908 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.978889 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:15.981394 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.981374 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 18:07:15.981522 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.981409 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wxsxp\"" Jul 23 18:07:15.982160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.982146 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 18:07:15.985975 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.985946 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 18:07:15.996499 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:15.996451 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l"] Jul 23 18:07:16.041237 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59026f15-d08a-4c2b-9f89-0e68e37ae323-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.041638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.041482 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzfs\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-kube-api-access-swzfs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.055172 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.055147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:16.066871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.066847 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g"] Jul 23 18:07:16.069719 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.069666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.076059 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.076039 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sxwzx\"" Jul 23 18:07:16.076206 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.076179 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 18:07:16.076503 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.076479 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 18:07:16.078989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.078970 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 18:07:16.091379 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.091357 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g"] Jul 23 18:07:16.142736 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.142662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.142736 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.142720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.142952 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.142850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.142952 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.142905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143057 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.142938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c4126-6271-4f5e-b2c7-10a22b234098-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143057 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143047 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143155 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143155 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143155 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143167 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59026f15-d08a-4c2b-9f89-0e68e37ae323-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzfs\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-kube-api-access-swzfs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143340 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9rg\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-kube-api-access-sk9rg\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.143645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143429 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143858 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.143939 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.143919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59026f15-d08a-4c2b-9f89-0e68e37ae323-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.145872 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.145851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.146006 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.145988 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.155324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.155291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.158194 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.158171 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzfs\" (UniqueName: \"kubernetes.io/projected/59026f15-d08a-4c2b-9f89-0e68e37ae323-kube-api-access-swzfs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f2m5l\" (UID: \"59026f15-d08a-4c2b-9f89-0e68e37ae323\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.233910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.233876 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p"] Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.243921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.243972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.244045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.244072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.244136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9rg\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-kube-api-access-sk9rg\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.244218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.245643 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.244270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c4126-6271-4f5e-b2c7-10a22b234098-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.246329 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.246258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.248910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.248498 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.248910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.248519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.248910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.248770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.248910 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.248792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c4126-6271-4f5e-b2c7-10a22b234098-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.265174 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.265148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9rg\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-kube-api-access-sk9rg\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.270377 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.270353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c4126-6271-4f5e-b2c7-10a22b234098-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zl92g\" (UID: \"0c1c4126-6271-4f5e-b2c7-10a22b234098\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.289333 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.289306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:16.302977 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.302951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" event={"ID":"5245db9c-8eb0-49d6-8e24-35b8d9b79c12","Type":"ContainerStarted","Data":"5d32e5f5b4091938ef42e0aaeb09f7b9b8ed7b72dd1749f9f72439fc901be773"} Jul 23 18:07:16.388091 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.388062 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:16.446587 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.446561 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l"] Jul 23 18:07:16.449031 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:07:16.448982 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59026f15_d08a_4c2b_9f89_0e68e37ae323.slice/crio-d589b9265c5179fb82fce85534d69f16cc52c888a493fddb9b390833782f3c95 WatchSource:0}: Error finding container d589b9265c5179fb82fce85534d69f16cc52c888a493fddb9b390833782f3c95: Status 404 returned error can't find the container with id d589b9265c5179fb82fce85534d69f16cc52c888a493fddb9b390833782f3c95 Jul 23 18:07:16.551612 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:16.551538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g"] Jul 23 18:07:16.554202 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:07:16.554171 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1c4126_6271_4f5e_b2c7_10a22b234098.slice/crio-79a2265c5f7f3c4d9d8cb3086d4a6f429e52e186c53415c217b243d35f4b9ff9 WatchSource:0}: Error finding container 79a2265c5f7f3c4d9d8cb3086d4a6f429e52e186c53415c217b243d35f4b9ff9: Status 404 returned error can't find the container with id 79a2265c5f7f3c4d9d8cb3086d4a6f429e52e186c53415c217b243d35f4b9ff9 Jul 23 18:07:17.312718 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:17.312605 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" event={"ID":"0c1c4126-6271-4f5e-b2c7-10a22b234098","Type":"ContainerStarted","Data":"79a2265c5f7f3c4d9d8cb3086d4a6f429e52e186c53415c217b243d35f4b9ff9"} Jul 23 18:07:17.315962 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:17.315909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" event={"ID":"59026f15-d08a-4c2b-9f89-0e68e37ae323","Type":"ContainerStarted","Data":"d589b9265c5179fb82fce85534d69f16cc52c888a493fddb9b390833782f3c95"} Jul 23 18:07:19.965952 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:19.965910 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:07:19.966221 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:19.965988 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:07:19.966221 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:19.966021 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:07:19.968085 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:19.968052 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:07:19.968211 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:19.968110 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:07:20.332741 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:20.332632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" event={"ID":"0c1c4126-6271-4f5e-b2c7-10a22b234098","Type":"ContainerStarted","Data":"23e350fdabfee7f252f9e6795ef7806445b20aca8e7633fc7f797120c7f4c54b"} Jul 23 18:07:20.332914 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:20.332776 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:20.334443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:20.334405 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" event={"ID":"59026f15-d08a-4c2b-9f89-0e68e37ae323","Type":"ContainerStarted","Data":"998d973ee552067aca76ecb64c5d08a59dcc2d378d685181ec879e4558e1c090"} Jul 23 18:07:20.444483 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:20.444420 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" podStartSLOduration=1.032670797 podStartE2EDuration="4.444399532s" totalImagesPullingTime="3.411728735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:20.443415666 +0000 UTC m=+508.818078844" watchObservedRunningTime="2026-07-23 18:07:20.444399532 +0000 UTC m=+508.819062712" Jul 23 18:07:20.573316 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:20.573260 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" podStartSLOduration=2.058816389 podStartE2EDuration="5.573240205s" totalImagesPullingTime="3.514423816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:20.571541923 +0000 UTC m=+508.946205114" watchObservedRunningTime="2026-07-23 18:07:20.573240205 +0000 UTC m=+508.947903384" Jul 23 18:07:21.290486 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.290437 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:21.295747 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.295723 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:21.340253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.340187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:21.341289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.341261 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f2m5l" Jul 23 18:07:21.341762 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.341739 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zl92g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:07:21.341870 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:21.341810 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" podUID="0c1c4126-6271-4f5e-b2c7-10a22b234098" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:07:24.340897 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:24.340870 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zl92g" Jul 23 18:07:30.382374 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:30.382341 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" event={"ID":"5245db9c-8eb0-49d6-8e24-35b8d9b79c12","Type":"ContainerStarted","Data":"4854abc8ecc075235e25af785adf5ce672b0dd25ce10325dcf6da1e3b8cf0927"} Jul 23 18:07:30.382846 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:30.382473 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:30.413000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:30.412945 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" podStartSLOduration=1.658956532 podStartE2EDuration="15.412927749s" totalImagesPullingTime="13.753971217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:30.409867453 +0000 UTC m=+518.784530635" watchObservedRunningTime="2026-07-23 18:07:30.412927749 +0000 UTC m=+518.787590928" Jul 23 18:07:32.308391 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.308359 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64646bd657-pnzs5"] Jul 23 18:07:32.311746 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.311728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.333594 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.333561 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64646bd657-pnzs5"] Jul 23 18:07:32.500904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.500868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hd56\" (UniqueName: \"kubernetes.io/projected/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-kube-api-access-6hd56\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.500904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.500907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.501109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.500922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.501109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.500946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-trusted-ca-bundle\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.501109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.501081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-oauth-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.501210 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.501112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-service-ca\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.501210 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.501130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-oauth-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.601921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.601842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hd56\" (UniqueName: \"kubernetes.io/projected/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-kube-api-access-6hd56\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.601921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.601880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.601921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.601895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.601921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.601916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-trusted-ca-bundle\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-oauth-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-service-ca\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-oauth-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602717 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602665 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602850 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602828 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-oauth-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602895 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-service-ca\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.602936 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.602886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-trusted-ca-bundle\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.604411 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.604384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-serving-cert\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.604506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.604409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-console-oauth-config\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.613090 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.613066 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hd56\" (UniqueName: \"kubernetes.io/projected/e19b9d3d-9f18-4ec2-b59c-6adecc3ec995-kube-api-access-6hd56\") pod \"console-64646bd657-pnzs5\" (UID: \"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995\") " pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.620952 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.620931 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:32.752877 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:32.752850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64646bd657-pnzs5"] Jul 23 18:07:32.755430 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:07:32.755405 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19b9d3d_9f18_4ec2_b59c_6adecc3ec995.slice/crio-985a07fb39383ba5890f58dfc166dd14f797438306b6f6d6e2317febd1898f1e WatchSource:0}: Error finding container 985a07fb39383ba5890f58dfc166dd14f797438306b6f6d6e2317febd1898f1e: Status 404 returned error can't find the container with id 985a07fb39383ba5890f58dfc166dd14f797438306b6f6d6e2317febd1898f1e Jul 23 18:07:33.396421 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:33.396386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64646bd657-pnzs5" event={"ID":"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995","Type":"ContainerStarted","Data":"010cdb737611022245d77b04ce87c31adbf4dd226027aa609c729bfa00cb549c"} Jul 23 18:07:33.396867 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:33.396438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64646bd657-pnzs5" event={"ID":"e19b9d3d-9f18-4ec2-b59c-6adecc3ec995","Type":"ContainerStarted","Data":"985a07fb39383ba5890f58dfc166dd14f797438306b6f6d6e2317febd1898f1e"} Jul 23 18:07:33.429372 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:33.429320 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64646bd657-pnzs5" podStartSLOduration=1.429304321 podStartE2EDuration="1.429304321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:33.425721221 +0000 UTC m=+521.800384393" watchObservedRunningTime="2026-07-23 18:07:33.429304321 +0000 UTC m=+521.803967501" Jul 23 18:07:41.389011 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:41.388981 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7ff5b44f4d-2rp6p" Jul 23 18:07:42.622180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:42.622140 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:42.622597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:42.622211 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:42.626944 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:42.626918 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:43.443451 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:43.443421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64646bd657-pnzs5" Jul 23 18:07:43.501606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:07:43.501576 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:08:08.522177 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.522070 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-664fccfcb7-2w7s6" podUID="10ab2643-7a46-4201-833d-d60fd8ddf38d" containerName="console" containerID="cri-o://af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea" gracePeriod=15 Jul 23 18:08:08.770276 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.770249 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:08:08.926280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926249 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926516 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926324 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkjzg\" (UniqueName: \"kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926580 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926507 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926633 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926575 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926711 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926651 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config" (OuterVolumeSpecName: "console-config") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:08.926711 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926830 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926721 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca\") pod \"10ab2643-7a46-4201-833d-d60fd8ddf38d\" (UID: \"10ab2643-7a46-4201-833d-d60fd8ddf38d\") " Jul 23 18:08:08.926941 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.926715 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:08.927069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.927014 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:08.927145 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.927080 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca" (OuterVolumeSpecName: "service-ca") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:08.927145 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.927100 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-trusted-ca-bundle\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:08.927145 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.927119 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:08.927145 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.927135 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-oauth-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:08.928582 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.928558 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:08:08.928582 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.928576 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:08:08.928752 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:08.928564 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg" (OuterVolumeSpecName: "kube-api-access-nkjzg") pod "10ab2643-7a46-4201-833d-d60fd8ddf38d" (UID: "10ab2643-7a46-4201-833d-d60fd8ddf38d"). InnerVolumeSpecName "kube-api-access-nkjzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:08:09.028279 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.028230 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-serving-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:09.028279 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.028276 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab2643-7a46-4201-833d-d60fd8ddf38d-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:09.028279 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.028287 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjzg\" (UniqueName: \"kubernetes.io/projected/10ab2643-7a46-4201-833d-d60fd8ddf38d-kube-api-access-nkjzg\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:09.028279 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.028297 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10ab2643-7a46-4201-833d-d60fd8ddf38d-console-oauth-config\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:08:09.555859 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.555823 2568 generic.go:356] "Generic (PLEG): container finished" podID="10ab2643-7a46-4201-833d-d60fd8ddf38d" containerID="af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea" exitCode=0 Jul 23 18:08:09.556300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.555863 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664fccfcb7-2w7s6" Jul 23 18:08:09.556300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.555899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664fccfcb7-2w7s6" event={"ID":"10ab2643-7a46-4201-833d-d60fd8ddf38d","Type":"ContainerDied","Data":"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea"} Jul 23 18:08:09.556300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.555946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664fccfcb7-2w7s6" event={"ID":"10ab2643-7a46-4201-833d-d60fd8ddf38d","Type":"ContainerDied","Data":"892fb75bddabe404d2e3815558cf06a17f86e78bdfc0fb7d4b3916204744300b"} Jul 23 18:08:09.556300 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.555965 2568 scope.go:117] "RemoveContainer" containerID="af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea" Jul 23 18:08:09.564658 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.564640 2568 scope.go:117] "RemoveContainer" containerID="af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea" Jul 23 18:08:09.564952 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:08:09.564934 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea\": container with ID starting with af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea not found: ID does not exist" containerID="af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea" Jul 23 18:08:09.565015 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.564960 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea"} err="failed to get container status \"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea\": rpc error: code = NotFound desc = could not find container \"af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea\": container with ID starting with af21516b717f06d9b0c80991518faa8910c6655d0bf1825dcb9ea655d081a4ea not found: ID does not exist" Jul 23 18:08:09.582218 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.582187 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:08:09.585104 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:09.585081 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-664fccfcb7-2w7s6"] Jul 23 18:08:10.152855 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:08:10.152814 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ab2643-7a46-4201-833d-d60fd8ddf38d" path="/var/lib/kubelet/pods/10ab2643-7a46-4201-833d-d60fd8ddf38d/volumes" Jul 23 18:09:35.146560 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.146523 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8"] Jul 23 18:09:35.147122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.146887 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ab2643-7a46-4201-833d-d60fd8ddf38d" containerName="console" Jul 23 18:09:35.147122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.146899 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ab2643-7a46-4201-833d-d60fd8ddf38d" containerName="console" Jul 23 18:09:35.147122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.146952 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ab2643-7a46-4201-833d-d60fd8ddf38d" containerName="console" Jul 23 18:09:35.148727 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.148708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.155656 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.155631 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4svxc\"" Jul 23 18:09:35.204701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.204646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8"] Jul 23 18:09:35.297615 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297615 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxb6\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-kube-api-access-tpxb6\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.297932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.298127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.298127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.297991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.398986 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.398890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.398986 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.398950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.398986 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.398979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxb6\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-kube-api-access-tpxb6\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399655 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399628 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399812 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399936 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.399973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.399954 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.401374 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.401354 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.401606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.401586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.408313 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.408292 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.408809 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.408789 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxb6\" (UniqueName: \"kubernetes.io/projected/74ab0959-51b3-4f2e-92f7-1dbea9160bd8-kube-api-access-tpxb6\") pod \"maas-default-gateway-openshift-default-56654dcf7-jl8j8\" (UID: \"74ab0959-51b3-4f2e-92f7-1dbea9160bd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.459491 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.459451 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:35.602364 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.602338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8"] Jul 23 18:09:35.604446 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:09:35.604410 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ab0959_51b3_4f2e_92f7_1dbea9160bd8.slice/crio-52092f8ad95382fa2a51cc225cfaed7b1ab1c577645ddb3c669cd6d6c7f2afb7 WatchSource:0}: Error finding container 52092f8ad95382fa2a51cc225cfaed7b1ab1c577645ddb3c669cd6d6c7f2afb7: Status 404 returned error can't find the container with id 52092f8ad95382fa2a51cc225cfaed7b1ab1c577645ddb3c669cd6d6c7f2afb7 Jul 23 18:09:35.606857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.606824 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:09:35.606924 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.606890 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:09:35.606924 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.606918 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:09:35.922821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.922697 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" event={"ID":"74ab0959-51b3-4f2e-92f7-1dbea9160bd8","Type":"ContainerStarted","Data":"6a8ef6fec5d654515266d5e290dc62fca4b4353b37e94ef7736f0690f58d03c1"} Jul 23 18:09:35.922821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.922767 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" event={"ID":"74ab0959-51b3-4f2e-92f7-1dbea9160bd8","Type":"ContainerStarted","Data":"52092f8ad95382fa2a51cc225cfaed7b1ab1c577645ddb3c669cd6d6c7f2afb7"} Jul 23 18:09:35.950135 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:35.950068 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" podStartSLOduration=0.950045503 podStartE2EDuration="950.045503ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:35.945938259 +0000 UTC m=+644.320601436" watchObservedRunningTime="2026-07-23 18:09:35.950045503 +0000 UTC m=+644.324708683" Jul 23 18:09:36.460274 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:36.460235 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:36.464987 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:36.464959 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:36.927071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:36.927035 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:09:36.928157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:09:36.928134 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:10:17.526906 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.526865 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:17.529541 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.529520 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.532175 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.532148 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 18:10:17.532331 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.532317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p7688\"" Jul 23 18:10:17.532388 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.532329 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 18:10:17.539796 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.539770 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:17.682934 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.682897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4g48\" (UniqueName: \"kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.683143 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.682951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.784495 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.784408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4g48\" (UniqueName: \"kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.784495 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.784460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.787127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.787093 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.793655 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.793614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4g48\" (UniqueName: \"kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48\") pod \"maas-controller-59bb95dbbf-srxv5\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.841795 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.841754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:17.977028 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:17.976996 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:17.979488 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:10:17.979456 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf594ef0_adaa_4bf6_b2fb_206f66cfa5d2.slice/crio-4bed03632df746ed8263debeb9237937e8679d4762e4219adaeb0bc921428cb7 WatchSource:0}: Error finding container 4bed03632df746ed8263debeb9237937e8679d4762e4219adaeb0bc921428cb7: Status 404 returned error can't find the container with id 4bed03632df746ed8263debeb9237937e8679d4762e4219adaeb0bc921428cb7 Jul 23 18:10:18.120104 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:18.120014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" event={"ID":"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2","Type":"ContainerStarted","Data":"4bed03632df746ed8263debeb9237937e8679d4762e4219adaeb0bc921428cb7"} Jul 23 18:10:21.138653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:21.138615 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" event={"ID":"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2","Type":"ContainerStarted","Data":"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714"} Jul 23 18:10:21.139091 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:21.138815 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:21.194348 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:21.194296 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" podStartSLOduration=1.732050334 podStartE2EDuration="4.194279173s" totalImagesPullingTime="2.462228839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:21.186826528 +0000 UTC m=+689.561489707" watchObservedRunningTime="2026-07-23 18:10:21.194279173 +0000 UTC m=+689.568942353" Jul 23 18:10:31.648917 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:31.648879 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:31.649392 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:31.649131 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" podUID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" containerName="manager" containerID="cri-o://b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714" gracePeriod=30 Jul 23 18:10:31.651035 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:31.651013 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:31.897473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:31.897444 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:32.019740 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.019626 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4g48\" (UniqueName: \"kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48\") pod \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " Jul 23 18:10:32.019740 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.019719 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert\") pod \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\" (UID: \"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2\") " Jul 23 18:10:32.021892 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.021864 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48" (OuterVolumeSpecName: "kube-api-access-w4g48") pod "bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" (UID: "bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2"). InnerVolumeSpecName "kube-api-access-w4g48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:10:32.021981 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.021900 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" (UID: "bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:10:32.121054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.121015 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4g48\" (UniqueName: \"kubernetes.io/projected/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-kube-api-access-w4g48\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:10:32.121054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.121047 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2-webhook-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:10:32.188555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.188516 2568 generic.go:356] "Generic (PLEG): container finished" podID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" containerID="b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714" exitCode=0 Jul 23 18:10:32.188712 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.188565 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" Jul 23 18:10:32.188712 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.188593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" event={"ID":"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2","Type":"ContainerDied","Data":"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714"} Jul 23 18:10:32.188712 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.188648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bb95dbbf-srxv5" event={"ID":"bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2","Type":"ContainerDied","Data":"4bed03632df746ed8263debeb9237937e8679d4762e4219adaeb0bc921428cb7"} Jul 23 18:10:32.188712 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.188671 2568 scope.go:117] "RemoveContainer" containerID="b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714" Jul 23 18:10:32.197427 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.197405 2568 scope.go:117] "RemoveContainer" containerID="b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714" Jul 23 18:10:32.197696 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:10:32.197663 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714\": container with ID starting with b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714 not found: ID does not exist" containerID="b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714" Jul 23 18:10:32.197770 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.197711 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714"} err="failed to get container status \"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714\": rpc error: code = NotFound desc = could not find container \"b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714\": container with ID starting with b2458d9dcd6c8a450de46a0ed163394bff6fcead9a81a4872a3cb66e91e0c714 not found: ID does not exist" Jul 23 18:10:32.215364 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.215330 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:32.219568 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:32.219546 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59bb95dbbf-srxv5"] Jul 23 18:10:34.152510 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:34.152473 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" path="/var/lib/kubelet/pods/bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2/volumes" Jul 23 18:10:51.466082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.465992 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx"] Jul 23 18:10:51.466662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.466491 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" containerName="manager" Jul 23 18:10:51.466662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.466511 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" containerName="manager" Jul 23 18:10:51.466662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.466605 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf594ef0-adaa-4bf6-b2fb-206f66cfa5d2" containerName="manager" Jul 23 18:10:51.468537 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.468515 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.470983 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.470963 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 23 18:10:51.471774 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.471755 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\"" Jul 23 18:10:51.472185 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.472154 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 18:10:51.472286 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.472157 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 18:10:51.488549 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.488516 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx"] Jul 23 18:10:51.607781 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.607674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld4g8\" (UniqueName: \"kubernetes.io/projected/448b23cd-aa05-407b-97b8-915e2be17994-kube-api-access-ld4g8\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.607964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.607846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b23cd-aa05-407b-97b8-915e2be17994-maas-api-tls\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.607964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.607921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b23cd-aa05-407b-97b8-915e2be17994-openshift-service-ca\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.708827 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.708776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b23cd-aa05-407b-97b8-915e2be17994-openshift-service-ca\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.709020 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.708843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld4g8\" (UniqueName: \"kubernetes.io/projected/448b23cd-aa05-407b-97b8-915e2be17994-kube-api-access-ld4g8\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.709020 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.708899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b23cd-aa05-407b-97b8-915e2be17994-maas-api-tls\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.709562 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.709521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b23cd-aa05-407b-97b8-915e2be17994-openshift-service-ca\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.711330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.711310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b23cd-aa05-407b-97b8-915e2be17994-maas-api-tls\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.720123 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.720067 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld4g8\" (UniqueName: \"kubernetes.io/projected/448b23cd-aa05-407b-97b8-915e2be17994-kube-api-access-ld4g8\") pod \"maas-api-684cc7d4cf-d6jnx\" (UID: \"448b23cd-aa05-407b-97b8-915e2be17994\") " pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.779389 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.779342 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:51.922000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:51.921889 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx"] Jul 23 18:10:51.924632 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:10:51.924608 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448b23cd_aa05_407b_97b8_915e2be17994.slice/crio-4a9571ec5984801104c6623001fbee16b155b4a25bccea4e6094a271124cb028 WatchSource:0}: Error finding container 4a9571ec5984801104c6623001fbee16b155b4a25bccea4e6094a271124cb028: Status 404 returned error can't find the container with id 4a9571ec5984801104c6623001fbee16b155b4a25bccea4e6094a271124cb028 Jul 23 18:10:52.283669 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:52.283636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" event={"ID":"448b23cd-aa05-407b-97b8-915e2be17994","Type":"ContainerStarted","Data":"4a9571ec5984801104c6623001fbee16b155b4a25bccea4e6094a271124cb028"} Jul 23 18:10:54.310888 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:54.310847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" event={"ID":"448b23cd-aa05-407b-97b8-915e2be17994","Type":"ContainerStarted","Data":"b171f9365a9bded7688e79339dfa3359fb035d236e2eddba517530ec950af209"} Jul 23 18:10:54.311354 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:54.311006 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:10:54.343535 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:10:54.343482 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" podStartSLOduration=1.828846758 podStartE2EDuration="3.343462296s" totalImagesPullingTime="1.514615538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:54.342433845 +0000 UTC m=+722.717097024" watchObservedRunningTime="2026-07-23 18:10:54.343462296 +0000 UTC m=+722.718125475" Jul 23 18:11:00.321238 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.321208 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-684cc7d4cf-d6jnx" Jul 23 18:11:00.339263 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.339231 2568 generic.go:356] "Generic (PLEG): container finished" podID="74ab0959-51b3-4f2e-92f7-1dbea9160bd8" containerID="6a8ef6fec5d654515266d5e290dc62fca4b4353b37e94ef7736f0690f58d03c1" exitCode=0 Jul 23 18:11:00.339421 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.339278 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" event={"ID":"74ab0959-51b3-4f2e-92f7-1dbea9160bd8","Type":"ContainerDied","Data":"6a8ef6fec5d654515266d5e290dc62fca4b4353b37e94ef7736f0690f58d03c1"} Jul 23 18:11:00.339641 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.339627 2568 scope.go:117] "RemoveContainer" containerID="6a8ef6fec5d654515266d5e290dc62fca4b4353b37e94ef7736f0690f58d03c1" Jul 23 18:11:00.340745 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.340712 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:11:00.340843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.340783 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:11:00.340843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:00.340816 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:11:01.345522 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:01.345483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" event={"ID":"74ab0959-51b3-4f2e-92f7-1dbea9160bd8","Type":"ContainerStarted","Data":"9d66919077f716f94fd4499173972f20a876f84f861e8249fbe4427ed457c5dc"} Jul 23 18:11:01.460653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:01.460609 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:11:01.460886 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:01.460864 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jl8j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 23 18:11:01.460932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:01.460913 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" podUID="74ab0959-51b3-4f2e-92f7-1dbea9160bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 23 18:11:02.460540 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:02.460500 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jl8j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 23 18:11:02.461016 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:02.460572 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" podUID="74ab0959-51b3-4f2e-92f7-1dbea9160bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 23 18:11:03.460312 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:03.460272 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jl8j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 23 18:11:03.460493 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:03.460342 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" podUID="74ab0959-51b3-4f2e-92f7-1dbea9160bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 23 18:11:04.460900 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:04.460861 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jl8j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 23 18:11:04.461320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:04.460919 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" podUID="74ab0959-51b3-4f2e-92f7-1dbea9160bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 23 18:11:05.464556 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:05.464524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:11:05.464959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:05.464803 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:11:05.465576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:05.465559 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jl8j8" Jul 23 18:11:15.574659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.574624 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4"] Jul 23 18:11:15.577209 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.577188 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.580422 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.580398 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 18:11:15.581563 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.581532 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 23 18:11:15.581660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.581542 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 18:11:15.581660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.581626 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qc4rc\"" Jul 23 18:11:15.591546 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.591521 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4"] Jul 23 18:11:15.647428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-kube-api-access-twl2j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.647428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.647725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.647725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.647725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647598 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.647725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.647622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749103 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-kube-api-access-twl2j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749373 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749592 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749592 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749563 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.749672 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.749653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.751416 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.751390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.751705 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.751666 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.764773 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.764751 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl2j\" (UniqueName: \"kubernetes.io/projected/15cfff98-e7e2-4e0f-ba45-23f5d358eda8-kube-api-access-twl2j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4\" (UID: \"15cfff98-e7e2-4e0f-ba45-23f5d358eda8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:15.887793 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:15.887752 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:16.032277 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:16.032246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4"] Jul 23 18:11:16.035808 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:11:16.035781 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cfff98_e7e2_4e0f_ba45_23f5d358eda8.slice/crio-ceb8847791ee98bc27e67cccf7d29cab4755daad00c360274694ca9e3627ec2a WatchSource:0}: Error finding container ceb8847791ee98bc27e67cccf7d29cab4755daad00c360274694ca9e3627ec2a: Status 404 returned error can't find the container with id ceb8847791ee98bc27e67cccf7d29cab4755daad00c360274694ca9e3627ec2a Jul 23 18:11:16.037248 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:16.037227 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:11:16.410153 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:16.410120 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" event={"ID":"15cfff98-e7e2-4e0f-ba45-23f5d358eda8","Type":"ContainerStarted","Data":"ceb8847791ee98bc27e67cccf7d29cab4755daad00c360274694ca9e3627ec2a"} Jul 23 18:11:17.417365 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:17.417328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" event={"ID":"15cfff98-e7e2-4e0f-ba45-23f5d358eda8","Type":"ContainerStarted","Data":"2e9d1e933377467b59bca832230c3d90b42ee66094e00a8a80ea4e638ce8ca96"} Jul 23 18:11:17.449187 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:17.449129 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" podStartSLOduration=1.166498411 podStartE2EDuration="2.449112913s" totalImagesPullingTime="1.282614502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:17.446329848 +0000 UTC m=+745.820993030" watchObservedRunningTime="2026-07-23 18:11:17.449112913 +0000 UTC m=+745.823776092" Jul 23 18:11:25.888591 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:25.888545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:25.889202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:25.888745 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:25.901037 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:25.901007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:26.472669 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:26.472638 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4" Jul 23 18:11:45.859222 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.859179 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk"] Jul 23 18:11:45.861946 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.861921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.864520 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.864499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 18:11:45.871645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.871612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk"] Jul 23 18:11:45.926858 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.926820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.927051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.926864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.927051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.926952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.927051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.926984 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwch\" (UniqueName: \"kubernetes.io/projected/305daefc-12e3-4020-9f71-4dfdc1b15898-kube-api-access-grwch\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.927227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.927083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305daefc-12e3-4020-9f71-4dfdc1b15898-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:45.927227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:45.927174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.027884 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.027837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.027884 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.027880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwch\" (UniqueName: \"kubernetes.io/projected/305daefc-12e3-4020-9f71-4dfdc1b15898-kube-api-access-grwch\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.027942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305daefc-12e3-4020-9f71-4dfdc1b15898-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.027963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.028013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.028036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.028391 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028483 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.028456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.028542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.028498 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.030257 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.030238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305daefc-12e3-4020-9f71-4dfdc1b15898-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.030562 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.030546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305daefc-12e3-4020-9f71-4dfdc1b15898-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.040897 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.040864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwch\" (UniqueName: \"kubernetes.io/projected/305daefc-12e3-4020-9f71-4dfdc1b15898-kube-api-access-grwch\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk\" (UID: \"305daefc-12e3-4020-9f71-4dfdc1b15898\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.173247 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.173209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:46.305570 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.305542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk"] Jul 23 18:11:46.307790 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:11:46.307752 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305daefc_12e3_4020_9f71_4dfdc1b15898.slice/crio-5e1136128b9002a969d190d4c0a9eed70bd13ef9454f0054e4720ef9c55a48c5 WatchSource:0}: Error finding container 5e1136128b9002a969d190d4c0a9eed70bd13ef9454f0054e4720ef9c55a48c5: Status 404 returned error can't find the container with id 5e1136128b9002a969d190d4c0a9eed70bd13ef9454f0054e4720ef9c55a48c5 Jul 23 18:11:46.556966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.556929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" event={"ID":"305daefc-12e3-4020-9f71-4dfdc1b15898","Type":"ContainerStarted","Data":"18eb5125f3e868cca19b554e97917638a305608f7486a5dd54e69e8a00265346"} Jul 23 18:11:46.557154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.556986 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" event={"ID":"305daefc-12e3-4020-9f71-4dfdc1b15898","Type":"ContainerStarted","Data":"5e1136128b9002a969d190d4c0a9eed70bd13ef9454f0054e4720ef9c55a48c5"} Jul 23 18:11:46.577095 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:46.577044 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" podStartSLOduration=1.416114039 podStartE2EDuration="1.577025881s" totalImagesPullingTime="160.911842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:46.575411229 +0000 UTC m=+774.950074422" watchObservedRunningTime="2026-07-23 18:11:46.577025881 +0000 UTC m=+774.951689061" Jul 23 18:11:56.173478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:56.173433 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:56.174157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:56.173511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:56.186223 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:56.186185 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:11:56.618197 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:11:56.618114 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk" Jul 23 18:28:12.446389 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.446354 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:28:12.454330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.454306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.457646 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.457618 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d2d0218-openshift-default-dockercfg-5jcmq\"" Jul 23 18:28:12.483456 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.483420 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:28:12.603208 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603208 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqjs\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603430 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603590 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.603590 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.603546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705068 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.704964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705068 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705272 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705405 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqjs\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705497 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705497 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705600 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705600 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705739 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705808 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705867 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705823 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.705932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.705854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.707847 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.707815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.707989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.707971 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.718395 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.718370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.718512 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.718413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqjs\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs\") pod \"e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.765322 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.765286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:12.910572 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.910546 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:28:12.913348 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:28:12.913318 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70e7128_aa9e_49c8_acf3_054669244c7a.slice/crio-6aff924e77e55700ff4876d351a7afbc899aa94b7ef51df310963db4ad0a33db WatchSource:0}: Error finding container 6aff924e77e55700ff4876d351a7afbc899aa94b7ef51df310963db4ad0a33db: Status 404 returned error can't find the container with id 6aff924e77e55700ff4876d351a7afbc899aa94b7ef51df310963db4ad0a33db Jul 23 18:28:12.915354 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.915319 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:12.915459 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.915391 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:12.915459 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:12.915424 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:13.077226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:13.077191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerStarted","Data":"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381"} Jul 23 18:28:13.077409 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:13.077250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerStarted","Data":"6aff924e77e55700ff4876d351a7afbc899aa94b7ef51df310963db4ad0a33db"} Jul 23 18:28:13.099732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:13.099660 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podStartSLOduration=1.099637135 podStartE2EDuration="1.099637135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:28:13.098595429 +0000 UTC m=+1761.473258610" watchObservedRunningTime="2026-07-23 18:28:13.099637135 +0000 UTC m=+1761.474300313" Jul 23 18:28:13.766147 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:13.766105 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:13.771794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:13.771768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:14.082429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:14.082350 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:14.083502 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:14.083480 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:29.790083 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:29.789275 2568 patch_prober.go:28] interesting pod/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:52410->10.132.0.37:15021: read: connection reset by peer" start-of-body= Jul 23 18:28:29.790083 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:29.789356 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:52410->10.132.0.37:15021: read: connection reset by peer" Jul 23 18:28:30.161171 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.161139 2568 generic.go:356] "Generic (PLEG): container finished" podID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerID="d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381" exitCode=0 Jul 23 18:28:30.161348 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.161209 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerDied","Data":"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381"} Jul 23 18:28:30.161597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.161582 2568 scope.go:117] "RemoveContainer" containerID="d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381" Jul 23 18:28:30.162638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.162613 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:30.162723 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.162709 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:30.162785 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:30.162744 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:31.169652 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:31.169612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerStarted","Data":"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74"} Jul 23 18:28:31.765746 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:31.765704 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:31.765932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:31.765900 2568 patch_prober.go:28] interesting pod/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:28:31.765980 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:31.765957 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 23 18:28:32.766298 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:32.766260 2568 patch_prober.go:28] interesting pod/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:28:32.766771 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:32.766329 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 23 18:28:33.765878 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:33.765835 2568 patch_prober.go:28] interesting pod/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:28:33.766056 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:33.765892 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 23 18:28:34.766121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:34.766081 2568 patch_prober.go:28] interesting pod/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:28:34.766607 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:34.766146 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 23 18:28:35.770340 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:35.770304 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:35.770826 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:35.770576 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:35.771404 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:35.771382 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:28:55.285775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:55.285741 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:28:55.286320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:55.285975 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" containerID="cri-o://e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74" gracePeriod=30 Jul 23 18:28:56.347672 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.346850 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:28:56.351130 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.351107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.353442 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.353414 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7b1deb3b-gw-openshift-default-dockercfg-wffqc\"" Jul 23 18:28:56.360661 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.360638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.430744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.430814 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.430864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.430914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.430994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.431040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw4s\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.431096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.431149 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.431296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.431177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532113 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532268 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw4s\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532596 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532596 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532596 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532596 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532518 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532596 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532534 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532828 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.532977 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.532960 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.533042 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.533022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.534620 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.534596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.534846 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.534827 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.549325 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.549295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw4s\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.550008 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.549987 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token\") pod \"e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.663700 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.663661 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:56.808290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.808231 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:28:56.810550 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:28:56.810516 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9641786c_3197_4beb_b2cf_97c6007dbd02.slice/crio-fb51d6255428754422e30f5f8650a8798caba2a74636d07ba8142ec5afbfd3d7 WatchSource:0}: Error finding container fb51d6255428754422e30f5f8650a8798caba2a74636d07ba8142ec5afbfd3d7: Status 404 returned error can't find the container with id fb51d6255428754422e30f5f8650a8798caba2a74636d07ba8142ec5afbfd3d7 Jul 23 18:28:56.812809 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.812778 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:56.812892 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.812842 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:56.812892 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:56.812873 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:28:57.296498 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:57.296460 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerStarted","Data":"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b"} Jul 23 18:28:57.296704 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:57.296526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerStarted","Data":"fb51d6255428754422e30f5f8650a8798caba2a74636d07ba8142ec5afbfd3d7"} Jul 23 18:28:57.321714 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:57.321645 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podStartSLOduration=1.321628735 podStartE2EDuration="1.321628735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:28:57.320332164 +0000 UTC m=+1805.694995347" watchObservedRunningTime="2026-07-23 18:28:57.321628735 +0000 UTC m=+1805.696291913" Jul 23 18:28:57.664658 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:57.664625 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:57.669720 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:57.669679 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:58.300974 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:58.300936 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:28:58.301760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:28:58.301735 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:00.532126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.532076 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:29:00.570665 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570595 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570665 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570637 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qqjs\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570705 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570749 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570790 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570823 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.570912 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570885 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.571163 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570919 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:00.571163 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570943 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.571163 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.570985 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs\") pod \"e70e7128-aa9e-49c8-acf3-054669244c7a\" (UID: \"e70e7128-aa9e-49c8-acf3-054669244c7a\") " Jul 23 18:29:00.571384 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.571361 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:29:00.571443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.571372 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.571443 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.571402 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:00.571697 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.571649 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:00.571794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.571749 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:00.573253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.573225 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:00.573253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.573224 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:29:00.573381 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.573358 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:29:00.574046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.574024 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs" (OuterVolumeSpecName: "kube-api-access-2qqjs") pod "e70e7128-aa9e-49c8-acf3-054669244c7a" (UID: "e70e7128-aa9e-49c8-acf3-054669244c7a"). InnerVolumeSpecName "kube-api-access-2qqjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:29:00.672469 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672433 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672469 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672463 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672469 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672477 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672489 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqjs\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-kube-api-access-2qqjs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672503 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672514 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672523 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70e7128-aa9e-49c8-acf3-054669244c7a-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:00.672737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:00.672533 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70e7128-aa9e-49c8-acf3-054669244c7a-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:01.315414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.315381 2568 generic.go:356] "Generic (PLEG): container finished" podID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerID="e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74" exitCode=0 Jul 23 18:29:01.315586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.315433 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" Jul 23 18:29:01.315586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.315469 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerDied","Data":"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74"} Jul 23 18:29:01.315586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.315530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q" event={"ID":"e70e7128-aa9e-49c8-acf3-054669244c7a","Type":"ContainerDied","Data":"6aff924e77e55700ff4876d351a7afbc899aa94b7ef51df310963db4ad0a33db"} Jul 23 18:29:01.315586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.315560 2568 scope.go:117] "RemoveContainer" containerID="e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74" Jul 23 18:29:01.324515 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.324499 2568 scope.go:117] "RemoveContainer" containerID="d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381" Jul 23 18:29:01.332254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.332237 2568 scope.go:117] "RemoveContainer" containerID="e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74" Jul 23 18:29:01.332508 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:29:01.332489 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74\": container with ID starting with e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74 not found: ID does not exist" containerID="e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74" Jul 23 18:29:01.332576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.332517 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74"} err="failed to get container status \"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74\": rpc error: code = NotFound desc = could not find container \"e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74\": container with ID starting with e0a88433046ad8ba755a7ece603a63044c9def76896d13049d9c488329372e74 not found: ID does not exist" Jul 23 18:29:01.332576 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.332533 2568 scope.go:117] "RemoveContainer" containerID="d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381" Jul 23 18:29:01.332789 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:29:01.332772 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381\": container with ID starting with d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381 not found: ID does not exist" containerID="d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381" Jul 23 18:29:01.332828 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.332793 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381"} err="failed to get container status \"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381\": rpc error: code = NotFound desc = could not find container \"d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381\": container with ID starting with d4bd49643169c1ac5faff0d41f5891940ea5dd4bf8d69d29b14599d4ce9f4381 not found: ID does not exist" Jul 23 18:29:01.337307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.337282 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:29:01.341448 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:01.341427 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d2d0218-openshift-default-bdb885bd9-vw87q"] Jul 23 18:29:02.153124 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:02.153091 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" path="/var/lib/kubelet/pods/e70e7128-aa9e-49c8-acf3-054669244c7a/volumes" Jul 23 18:29:12.369025 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.368936 2568 generic.go:356] "Generic (PLEG): container finished" podID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerID="d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b" exitCode=0 Jul 23 18:29:12.369025 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.369008 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerDied","Data":"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b"} Jul 23 18:29:12.369478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.369438 2568 scope.go:117] "RemoveContainer" containerID="d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b" Jul 23 18:29:12.370499 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.370469 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:12.370604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.370550 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:12.370604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:12.370599 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:13.301783 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:13.301734 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:13.376468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:13.376427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerStarted","Data":"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5"} Jul 23 18:29:13.664194 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:13.664157 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:13.665425 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:13.665400 2568 patch_prober.go:28] interesting pod/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 18:29:13.665535 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:13.665452 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 18:29:14.664416 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:14.664371 2568 patch_prober.go:28] interesting pod/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 18:29:14.664972 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:14.664446 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 18:29:15.664711 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:15.664656 2568 patch_prober.go:28] interesting pod/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 18:29:15.665202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:15.664738 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 18:29:16.664857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:16.664817 2568 patch_prober.go:28] interesting pod/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 18:29:16.665257 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:16.664873 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 18:29:17.667958 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:17.667928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:17.668334 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:17.668316 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:17.669071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:17.669053 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:23.904238 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.904203 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26"] Jul 23 18:29:23.904621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.904572 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:23.904621 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.904583 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:23.904722 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.904643 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:23.908081 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.908065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:23.910813 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.910784 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:29:23.917479 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.917456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26"] Jul 23 18:29:23.998714 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:23.998648 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26\" (UID: \"e65d2ad0-573d-4a36-9ff1-056577581d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:24.099961 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:24.099924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26\" (UID: \"e65d2ad0-573d-4a36-9ff1-056577581d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:24.100519 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:24.100500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26\" (UID: \"e65d2ad0-573d-4a36-9ff1-056577581d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:24.218994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:24.218916 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:24.557980 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:24.557956 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26"] Jul 23 18:29:24.560550 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:29:24.560515 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65d2ad0_573d_4a36_9ff1_056577581d18.slice/crio-4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449 WatchSource:0}: Error finding container 4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449: Status 404 returned error can't find the container with id 4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449 Jul 23 18:29:24.562846 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:24.562828 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:29:25.441614 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:25.441578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" event={"ID":"e65d2ad0-573d-4a36-9ff1-056577581d18","Type":"ContainerStarted","Data":"4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449"} Jul 23 18:29:27.455659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:27.455618 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" event={"ID":"e65d2ad0-573d-4a36-9ff1-056577581d18","Type":"ContainerStarted","Data":"d497c718b02c8f2beaa8a897905954722aa1bd4ffc6218956c9aba3651536afa"} Jul 23 18:29:27.472103 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:27.472047 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" podStartSLOduration=2.008234008 podStartE2EDuration="4.472027372s" totalImagesPullingTime="2.463793364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:29:27.47134748 +0000 UTC m=+1835.846010660" watchObservedRunningTime="2026-07-23 18:29:27.472027372 +0000 UTC m=+1835.846690552" Jul 23 18:29:28.461766 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:28.461730 2568 generic.go:356] "Generic (PLEG): container finished" podID="e65d2ad0-573d-4a36-9ff1-056577581d18" containerID="d497c718b02c8f2beaa8a897905954722aa1bd4ffc6218956c9aba3651536afa" exitCode=0 Jul 23 18:29:28.462164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:28.461814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" event={"ID":"e65d2ad0-573d-4a36-9ff1-056577581d18","Type":"ContainerDied","Data":"d497c718b02c8f2beaa8a897905954722aa1bd4ffc6218956c9aba3651536afa"} Jul 23 18:29:29.600305 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:29.600276 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:29.659547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:29.659512 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca\") pod \"e65d2ad0-573d-4a36-9ff1-056577581d18\" (UID: \"e65d2ad0-573d-4a36-9ff1-056577581d18\") " Jul 23 18:29:29.659943 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:29.659916 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e65d2ad0-573d-4a36-9ff1-056577581d18" (UID: "e65d2ad0-573d-4a36-9ff1-056577581d18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:29:29.760428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:29.760344 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d2ad0-573d-4a36-9ff1-056577581d18-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:30.472339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:30.472252 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" Jul 23 18:29:30.472339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:30.472263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26" event={"ID":"e65d2ad0-573d-4a36-9ff1-056577581d18","Type":"ContainerDied","Data":"4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449"} Jul 23 18:29:30.472339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:30.472308 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4060f8e98d3cf6a51bbc989bd5bd68862548296d765130be0563f24b2eca7449" Jul 23 18:29:38.977187 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:38.977151 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26"] Jul 23 18:29:38.979574 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:38.979549 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b1deb3b-qxx26"] Jul 23 18:29:39.140079 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.140043 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:29:39.140305 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.140257 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" containerID="cri-o://6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5" gracePeriod=30 Jul 23 18:29:39.986081 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986040 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:29:39.986572 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986553 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:39.986879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986576 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:39.986879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986605 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65d2ad0-573d-4a36-9ff1-056577581d18" containerName="revoke-keys" Jul 23 18:29:39.986879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986615 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65d2ad0-573d-4a36-9ff1-056577581d18" containerName="revoke-keys" Jul 23 18:29:39.986879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986749 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70e7128-aa9e-49c8-acf3-054669244c7a" containerName="istio-proxy" Jul 23 18:29:39.986879 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.986763 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65d2ad0-573d-4a36-9ff1-056577581d18" containerName="revoke-keys" Jul 23 18:29:39.989927 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.989907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:39.992432 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:39.992414 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9efa58d0-openshift-default-dockercfg-m79wd\"" Jul 23 18:29:40.005491 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.005465 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:29:40.060039 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060190 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060190 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060190 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060190 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060161 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060190 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tp9\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.060468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.060337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.152971 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.152936 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65d2ad0-573d-4a36-9ff1-056577581d18" path="/var/lib/kubelet/pods/e65d2ad0-573d-4a36-9ff1-056577581d18/volumes" Jul 23 18:29:40.161308 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161395 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161534 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161598 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161724 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161727 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161765 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.161801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.162005 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.162005 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tp9\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.162005 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.161928 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.162156 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.162020 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.162462 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.162443 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.163752 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.163725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.164025 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.164006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.169589 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.169563 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.169707 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.169647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tp9\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9\") pod \"e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.300015 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.299923 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:40.442600 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.442575 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:29:40.444462 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:29:40.444436 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc9c7f6_82b7_4b64_a9a5_45cb0b5c880f.slice/crio-31775c325558241e8d33e3a9b5f03823e9d4dbcd84500c01588e7f2a2a200ae0 WatchSource:0}: Error finding container 31775c325558241e8d33e3a9b5f03823e9d4dbcd84500c01588e7f2a2a200ae0: Status 404 returned error can't find the container with id 31775c325558241e8d33e3a9b5f03823e9d4dbcd84500c01588e7f2a2a200ae0 Jul 23 18:29:40.446521 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.446487 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:40.446604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.446557 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:40.446604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.446584 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:40.521286 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.521252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerStarted","Data":"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274"} Jul 23 18:29:40.521439 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.521305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerStarted","Data":"31775c325558241e8d33e3a9b5f03823e9d4dbcd84500c01588e7f2a2a200ae0"} Jul 23 18:29:40.541736 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:40.541665 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podStartSLOduration=1.541647308 podStartE2EDuration="1.541647308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:29:40.540906089 +0000 UTC m=+1848.915569282" watchObservedRunningTime="2026-07-23 18:29:40.541647308 +0000 UTC m=+1848.916310542" Jul 23 18:29:41.300562 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:41.300523 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:41.305500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:41.305476 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:41.526437 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:41.526406 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:41.527235 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:41.527216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:44.395838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.395812 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:44.502646 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502542 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502646 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502673 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502753 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502797 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvw4s\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502835 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502857 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.502898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502892 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token\") pod \"9641786c-3197-4beb-b2cf-97c6007dbd02\" (UID: \"9641786c-3197-4beb-b2cf-97c6007dbd02\") " Jul 23 18:29:44.503258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.502955 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:44.503258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503118 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:29:44.503258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:44.503412 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503120 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:44.503412 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503332 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641786c-3197-4beb-b2cf-97c6007dbd02-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.503412 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503352 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.503412 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503365 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.503749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.503727 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data" (OuterVolumeSpecName: "istio-data") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:44.505119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.505083 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:29:44.505119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.505095 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s" (OuterVolumeSpecName: "kube-api-access-jvw4s") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "kube-api-access-jvw4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:29:44.505328 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.505309 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:29:44.505424 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.505409 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token" (OuterVolumeSpecName: "istio-token") pod "9641786c-3197-4beb-b2cf-97c6007dbd02" (UID: "9641786c-3197-4beb-b2cf-97c6007dbd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:29:44.541452 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.541418 2568 generic.go:356] "Generic (PLEG): container finished" podID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerID="6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5" exitCode=0 Jul 23 18:29:44.541604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.541458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerDied","Data":"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5"} Jul 23 18:29:44.541604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.541521 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" event={"ID":"9641786c-3197-4beb-b2cf-97c6007dbd02","Type":"ContainerDied","Data":"fb51d6255428754422e30f5f8650a8798caba2a74636d07ba8142ec5afbfd3d7"} Jul 23 18:29:44.541604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.541544 2568 scope.go:117] "RemoveContainer" containerID="6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5" Jul 23 18:29:44.541604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.541474 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx" Jul 23 18:29:44.550589 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.550571 2568 scope.go:117] "RemoveContainer" containerID="d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b" Jul 23 18:29:44.558444 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.558426 2568 scope.go:117] "RemoveContainer" containerID="6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5" Jul 23 18:29:44.558693 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:29:44.558660 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5\": container with ID starting with 6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5 not found: ID does not exist" containerID="6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5" Jul 23 18:29:44.558778 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.558710 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5"} err="failed to get container status \"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5\": rpc error: code = NotFound desc = could not find container \"6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5\": container with ID starting with 6d35876213479221b3f39225f652da1e30a670f01049510578e6566d5a834dc5 not found: ID does not exist" Jul 23 18:29:44.558778 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.558727 2568 scope.go:117] "RemoveContainer" containerID="d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b" Jul 23 18:29:44.558972 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:29:44.558950 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b\": container with ID starting with d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b not found: ID does not exist" containerID="d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b" Jul 23 18:29:44.559011 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.558980 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b"} err="failed to get container status \"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b\": rpc error: code = NotFound desc = could not find container \"d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b\": container with ID starting with d5adc4f086840809dc3d16535c4ff9c8f74b759417d39ce3ad3fe868df3e016b not found: ID does not exist" Jul 23 18:29:44.564422 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.564395 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:29:44.567480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.567457 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7b1deb3b-gw-openshift-default-5f4cb9d499-z67wx"] Jul 23 18:29:44.604099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604062 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.604099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604094 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.604099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604107 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw4s\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-kube-api-access-jvw4s\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.604324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604119 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.604324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604130 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:44.604324 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:44.604138 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641786c-3197-4beb-b2cf-97c6007dbd02-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:29:46.153173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:46.153140 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" path="/var/lib/kubelet/pods/9641786c-3197-4beb-b2cf-97c6007dbd02/volumes" Jul 23 18:29:56.528234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.528154 2568 patch_prober.go:28] interesting pod/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:29:56.528234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.528217 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 23 18:29:56.605610 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.605530 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerID="f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274" exitCode=0 Jul 23 18:29:56.605610 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.605583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerDied","Data":"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274"} Jul 23 18:29:56.606023 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.606007 2568 scope.go:117] "RemoveContainer" containerID="f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274" Jul 23 18:29:56.607173 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.607137 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:56.607298 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.607242 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:56.607369 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:56.607293 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:29:57.611902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:57.611866 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerStarted","Data":"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046"} Jul 23 18:29:58.300552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:58.300510 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:29:58.300780 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:58.300633 2568 patch_prober.go:28] interesting pod/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:29:58.300780 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:58.300677 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 23 18:29:59.300792 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:59.300747 2568 patch_prober.go:28] interesting pod/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:29:59.301195 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:29:59.300812 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 23 18:30:00.171974 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.171941 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq"] Jul 23 18:30:00.172290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.172279 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.172335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.172292 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.172375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.172366 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.172412 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.172378 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.174339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.174322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" Jul 23 18:30:00.178158 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.178128 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:30:00.189751 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.189726 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" Jul 23 18:30:00.210785 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.210746 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw"] Jul 23 18:30:00.211380 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.211355 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.211380 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.211381 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9641786c-3197-4beb-b2cf-97c6007dbd02" containerName="istio-proxy" Jul 23 18:30:00.214168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.214136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" Jul 23 18:30:00.214587 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.214568 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq"] Jul 23 18:30:00.220975 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.220947 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw"] Jul 23 18:30:00.231733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.231711 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" Jul 23 18:30:00.302493 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.302031 2568 patch_prober.go:28] interesting pod/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:30:00.302493 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.302092 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 23 18:30:00.386744 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.384667 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq"] Jul 23 18:30:00.410592 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:30:00.410565 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d7e4e7_2c2e_4bc6_ad42_5af28ba4319a.slice/crio-7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add WatchSource:0}: Error finding container 7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add: Status 404 returned error can't find the container with id 7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add Jul 23 18:30:00.431328 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.431306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw"] Jul 23 18:30:00.439732 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:30:00.439708 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4814fd81_a7ca_49c1_9c72_1a88284a185a.slice/crio-80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6 WatchSource:0}: Error finding container 80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6: Status 404 returned error can't find the container with id 80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6 Jul 23 18:30:00.632393 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.632362 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" event={"ID":"4814fd81-a7ca-49c1-9c72-1a88284a185a","Type":"ContainerStarted","Data":"80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6"} Jul 23 18:30:00.633474 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:00.633450 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" event={"ID":"38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a","Type":"ContainerStarted","Data":"7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add"} Jul 23 18:30:00.685779 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:30:00.685676 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d7e4e7_2c2e_4bc6_ad42_5af28ba4319a.slice/crio-b9c151db2a8749cead6080a55d20dfb4ccfe73f2330f1d72fd75284e8ee4be62.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:30:01.300656 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.300615 2568 patch_prober.go:28] interesting pod/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:30:01.300850 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.300700 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 23 18:30:01.639796 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.639755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" event={"ID":"4814fd81-a7ca-49c1-9c72-1a88284a185a","Type":"ContainerStarted","Data":"1fd6a8c7f103961992fabc97cc4badfa5228d3fa796d2dbae4303a79dec3fc4d"} Jul 23 18:30:01.641514 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.641479 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" event={"ID":"38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a","Type":"ContainerStarted","Data":"b9c151db2a8749cead6080a55d20dfb4ccfe73f2330f1d72fd75284e8ee4be62"} Jul 23 18:30:01.655227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.655182 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" podStartSLOduration=1.655168218 podStartE2EDuration="1.655168218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:01.653842648 +0000 UTC m=+1870.028505829" watchObservedRunningTime="2026-07-23 18:30:01.655168218 +0000 UTC m=+1870.029831397" Jul 23 18:30:01.668558 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:01.668510 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" podStartSLOduration=1.6684967739999998 podStartE2EDuration="1.668496774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:01.667342318 +0000 UTC m=+1870.042005498" watchObservedRunningTime="2026-07-23 18:30:01.668496774 +0000 UTC m=+1870.043159953" Jul 23 18:30:02.304211 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.304176 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:30:02.304477 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.304450 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:30:02.305246 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.305226 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:30:02.647719 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.647660 2568 generic.go:356] "Generic (PLEG): container finished" podID="4814fd81-a7ca-49c1-9c72-1a88284a185a" containerID="1fd6a8c7f103961992fabc97cc4badfa5228d3fa796d2dbae4303a79dec3fc4d" exitCode=0 Jul 23 18:30:02.648139 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.647739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" event={"ID":"4814fd81-a7ca-49c1-9c72-1a88284a185a","Type":"ContainerDied","Data":"1fd6a8c7f103961992fabc97cc4badfa5228d3fa796d2dbae4303a79dec3fc4d"} Jul 23 18:30:02.649194 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.649174 2568 generic.go:356] "Generic (PLEG): container finished" podID="38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a" containerID="b9c151db2a8749cead6080a55d20dfb4ccfe73f2330f1d72fd75284e8ee4be62" exitCode=0 Jul 23 18:30:02.649332 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:02.649267 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" event={"ID":"38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a","Type":"ContainerDied","Data":"b9c151db2a8749cead6080a55d20dfb4ccfe73f2330f1d72fd75284e8ee4be62"} Jul 23 18:30:03.824808 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:03.824782 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" Jul 23 18:30:03.828693 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:03.828658 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" Jul 23 18:30:04.661297 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.661270 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" Jul 23 18:30:04.661485 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.661269 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-c62fw" event={"ID":"4814fd81-a7ca-49c1-9c72-1a88284a185a","Type":"ContainerDied","Data":"80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6"} Jul 23 18:30:04.661485 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.661387 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ec8caefc516a3ab1ac67ba16db78e6c636443409c70d5b87f5f1e3870abaa6" Jul 23 18:30:04.662618 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.662598 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" Jul 23 18:30:04.662757 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.662633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq" event={"ID":"38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a","Type":"ContainerDied","Data":"7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add"} Jul 23 18:30:04.662757 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:04.662665 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2ba0ab7f355b7d5da3107cb918a35c3757df8ade2bd0e451d7b027dec04add" Jul 23 18:30:07.833327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833291 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n"] Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833643 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4814fd81-a7ca-49c1-9c72-1a88284a185a" containerName="cleanup" Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833654 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4814fd81-a7ca-49c1-9c72-1a88284a185a" containerName="cleanup" Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833669 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a" containerName="cleanup" Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833675 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a" containerName="cleanup" Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833747 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4814fd81-a7ca-49c1-9c72-1a88284a185a" containerName="cleanup" Jul 23 18:30:07.833821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.833757 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a" containerName="cleanup" Jul 23 18:30:07.835821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.835801 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:07.838123 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.838101 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:30:07.843928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.843826 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n"] Jul 23 18:30:07.936547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:07.936516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n\" (UID: \"de177e72-0167-41b5-9aa8-e8ed2ca62f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:08.037646 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.037606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n\" (UID: \"de177e72-0167-41b5-9aa8-e8ed2ca62f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:08.038267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.038246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n\" (UID: \"de177e72-0167-41b5-9aa8-e8ed2ca62f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:08.148162 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.148134 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:08.273153 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.273103 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n"] Jul 23 18:30:08.275985 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:30:08.275950 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde177e72_0167_41b5_9aa8_e8ed2ca62f88.slice/crio-bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278 WatchSource:0}: Error finding container bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278: Status 404 returned error can't find the container with id bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278 Jul 23 18:30:08.685447 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.685403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" event={"ID":"de177e72-0167-41b5-9aa8-e8ed2ca62f88","Type":"ContainerStarted","Data":"5cd852dc938fda9553f64c4c58a819ae012f745148acc0042025e8ed10952b1b"} Jul 23 18:30:08.685740 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.685510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" event={"ID":"de177e72-0167-41b5-9aa8-e8ed2ca62f88","Type":"ContainerStarted","Data":"bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278"} Jul 23 18:30:08.701749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:08.701699 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" podStartSLOduration=1.701666135 podStartE2EDuration="1.701666135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:08.699278663 +0000 UTC m=+1877.073941843" watchObservedRunningTime="2026-07-23 18:30:08.701666135 +0000 UTC m=+1877.076329315" Jul 23 18:30:09.693636 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:09.693449 2568 generic.go:356] "Generic (PLEG): container finished" podID="de177e72-0167-41b5-9aa8-e8ed2ca62f88" containerID="5cd852dc938fda9553f64c4c58a819ae012f745148acc0042025e8ed10952b1b" exitCode=0 Jul 23 18:30:09.693636 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:09.693575 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" event={"ID":"de177e72-0167-41b5-9aa8-e8ed2ca62f88","Type":"ContainerDied","Data":"5cd852dc938fda9553f64c4c58a819ae012f745148acc0042025e8ed10952b1b"} Jul 23 18:30:10.827893 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:10.827869 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:10.864054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:10.864026 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca\") pod \"de177e72-0167-41b5-9aa8-e8ed2ca62f88\" (UID: \"de177e72-0167-41b5-9aa8-e8ed2ca62f88\") " Jul 23 18:30:10.864339 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:10.864316 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de177e72-0167-41b5-9aa8-e8ed2ca62f88" (UID: "de177e72-0167-41b5-9aa8-e8ed2ca62f88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:30:10.965550 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:10.965462 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de177e72-0167-41b5-9aa8-e8ed2ca62f88-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:11.704169 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:11.704144 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" Jul 23 18:30:11.704367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:11.704143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n" event={"ID":"de177e72-0167-41b5-9aa8-e8ed2ca62f88","Type":"ContainerDied","Data":"bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278"} Jul 23 18:30:11.704367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:11.704265 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe35158765a9057a5baedf507574e5cbd233e4e509b1d3c66461b7098993278" Jul 23 18:30:18.275733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:18.275647 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq"] Jul 23 18:30:18.280331 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:18.280305 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9efa58d0-29747190-2s6rq"] Jul 23 18:30:20.159065 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:20.159024 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a" path="/var/lib/kubelet/pods/38d7e4e7-2c2e-4bc6-ad42-5af28ba4319a/volumes" Jul 23 18:30:22.906386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:22.906353 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n"] Jul 23 18:30:22.909101 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:22.909077 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9efa58d0-dwz7n"] Jul 23 18:30:24.108228 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.108196 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:30:24.108786 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.108396 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" containerID="cri-o://417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046" gracePeriod=30 Jul 23 18:30:24.152966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.152934 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de177e72-0167-41b5-9aa8-e8ed2ca62f88" path="/var/lib/kubelet/pods/de177e72-0167-41b5-9aa8-e8ed2ca62f88/volumes" Jul 23 18:30:24.901732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.901699 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:30:24.902052 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.902040 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de177e72-0167-41b5-9aa8-e8ed2ca62f88" containerName="revoke-keys" Jul 23 18:30:24.902052 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.902053 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="de177e72-0167-41b5-9aa8-e8ed2ca62f88" containerName="revoke-keys" Jul 23 18:30:24.902178 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.902144 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="de177e72-0167-41b5-9aa8-e8ed2ca62f88" containerName="revoke-keys" Jul 23 18:30:24.905116 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.905095 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:24.907390 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.907369 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b5680af2-openshift-default-dockercfg-b2dwx\"" Jul 23 18:30:24.916846 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:24.916822 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:30:25.001119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001391 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001391 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001391 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cbh\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.001552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.001488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cbh\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102578 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.102891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.102791 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.103154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.103130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.103284 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.103227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.104808 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.104783 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.105136 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.105115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.112239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.112195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cbh\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.112598 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.112297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token\") pod \"e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.219036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.218945 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:25.358180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.358157 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:30:25.360208 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:30:25.360182 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b055711_6260_447a_87e4_f3c2b932560a.slice/crio-be10db509938e9f6ae38c63cb94cfb8e065aab8e0d9e0f34a2842c2d34a3fe2e WatchSource:0}: Error finding container be10db509938e9f6ae38c63cb94cfb8e065aab8e0d9e0f34a2842c2d34a3fe2e: Status 404 returned error can't find the container with id be10db509938e9f6ae38c63cb94cfb8e065aab8e0d9e0f34a2842c2d34a3fe2e Jul 23 18:30:25.362357 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.362322 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:25.362456 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.362400 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:25.362456 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.362444 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:25.775902 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.775863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerStarted","Data":"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0"} Jul 23 18:30:25.776103 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.775945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerStarted","Data":"be10db509938e9f6ae38c63cb94cfb8e065aab8e0d9e0f34a2842c2d34a3fe2e"} Jul 23 18:30:25.795835 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:25.795784 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podStartSLOduration=1.7957679739999999 podStartE2EDuration="1.795767974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:25.795072145 +0000 UTC m=+1894.169735349" watchObservedRunningTime="2026-07-23 18:30:25.795767974 +0000 UTC m=+1894.170431202" Jul 23 18:30:26.219765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:26.219732 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:26.224674 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:26.224648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:26.781860 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:26.781831 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:26.782897 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:26.782875 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:29.363414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.363390 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:30:29.447672 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447638 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447741 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7tp9\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447763 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447780 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447807 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447830 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.447873 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447868 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.448249 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447924 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.448249 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447938 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:30:29.448249 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.447975 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs\") pod \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\" (UID: \"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f\") " Jul 23 18:30:29.448249 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448111 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:30:29.448463 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448293 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:30:29.448463 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448351 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.448463 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448352 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:30:29.448463 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448363 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.448831 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.448808 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:30:29.450160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.450128 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:30:29.450274 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.450205 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:30:29.450441 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.450419 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9" (OuterVolumeSpecName: "kube-api-access-r7tp9") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "kube-api-access-r7tp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:30:29.450532 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.450443 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" (UID: "ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:30:29.549555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549519 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tp9\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-kube-api-access-r7tp9\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549549 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549555 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549561 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549569 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549577 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549586 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.549820 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.549595 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:29.805130 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.805037 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerID="417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046" exitCode=0 Jul 23 18:30:29.805130 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.805104 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" Jul 23 18:30:29.805326 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.805128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerDied","Data":"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046"} Jul 23 18:30:29.805326 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.805215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb" event={"ID":"ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f","Type":"ContainerDied","Data":"31775c325558241e8d33e3a9b5f03823e9d4dbcd84500c01588e7f2a2a200ae0"} Jul 23 18:30:29.805326 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.805244 2568 scope.go:117] "RemoveContainer" containerID="417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046" Jul 23 18:30:29.815079 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.815064 2568 scope.go:117] "RemoveContainer" containerID="f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274" Jul 23 18:30:29.822773 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.822755 2568 scope.go:117] "RemoveContainer" containerID="417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046" Jul 23 18:30:29.823027 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:30:29.823008 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046\": container with ID starting with 417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046 not found: ID does not exist" containerID="417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046" Jul 23 18:30:29.823082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.823039 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046"} err="failed to get container status \"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046\": rpc error: code = NotFound desc = could not find container \"417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046\": container with ID starting with 417f65b9b8cb24977c92c6adcc16c9d64df67474a0526362233f410a808c2046 not found: ID does not exist" Jul 23 18:30:29.823082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.823061 2568 scope.go:117] "RemoveContainer" containerID="f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274" Jul 23 18:30:29.823285 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:30:29.823268 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274\": container with ID starting with f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274 not found: ID does not exist" containerID="f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274" Jul 23 18:30:29.823329 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.823288 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274"} err="failed to get container status \"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274\": rpc error: code = NotFound desc = could not find container \"f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274\": container with ID starting with f9354140b78124aff0ea748f0326f36df4a52de776d1a8fd98fea3f8c6af0274 not found: ID does not exist" Jul 23 18:30:29.828358 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.828340 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:30:29.832653 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:29.832634 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9efa58d0-openshift-default-6884b4f7f5-tz2gb"] Jul 23 18:30:30.153042 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:30.153001 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" path="/var/lib/kubelet/pods/ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f/volumes" Jul 23 18:30:38.851988 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.851956 2568 generic.go:356] "Generic (PLEG): container finished" podID="1b055711-6260-447a-87e4-f3c2b932560a" containerID="fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0" exitCode=0 Jul 23 18:30:38.852409 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.852021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerDied","Data":"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0"} Jul 23 18:30:38.852453 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.852411 2568 scope.go:117] "RemoveContainer" containerID="fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0" Jul 23 18:30:38.853420 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.853389 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:38.853492 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.853475 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:38.853542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:38.853527 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:30:39.858890 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:39.858858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerStarted","Data":"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8"} Jul 23 18:30:40.219125 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:40.219089 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:40.220264 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:40.220239 2568 patch_prober.go:28] interesting pod/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:30:40.220321 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:40.220300 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 23 18:30:41.219926 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:41.219882 2568 patch_prober.go:28] interesting pod/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:30:41.220334 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:41.219948 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 23 18:30:41.784948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:41.784791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:42.220111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:42.220070 2568 patch_prober.go:28] interesting pod/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:30:42.220515 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:42.220130 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 23 18:30:43.220200 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:43.220160 2568 patch_prober.go:28] interesting pod/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:30:43.220607 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:43.220220 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 23 18:30:44.223347 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:44.223317 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:44.224393 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:44.224374 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:30:52.110656 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.110620 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4"] Jul 23 18:30:52.111154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.111052 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:30:52.111154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.111066 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:30:52.111154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.111080 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:30:52.111154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.111086 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:30:52.111154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.111150 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:30:52.113598 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.113575 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:52.116450 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.116425 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:30:52.122306 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.122279 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4"] Jul 23 18:30:52.171330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.171299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4\" (UID: \"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:52.272102 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.272069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4\" (UID: \"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:52.272724 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.272702 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4\" (UID: \"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:52.426083 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.426044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:52.552656 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.552629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4"] Jul 23 18:30:52.554591 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:30:52.554557 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4afb1ec_e120_40a2_a1c0_e7b0406f1d54.slice/crio-370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e WatchSource:0}: Error finding container 370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e: Status 404 returned error can't find the container with id 370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e Jul 23 18:30:52.923524 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.923491 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" event={"ID":"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54","Type":"ContainerStarted","Data":"75efec8fc2fff5e066cee2146886395dfcafe362a516ae2efa116cbea08633a8"} Jul 23 18:30:52.923678 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.923546 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" event={"ID":"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54","Type":"ContainerStarted","Data":"370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e"} Jul 23 18:30:52.939448 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:52.939397 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" podStartSLOduration=0.939382365 podStartE2EDuration="939.382365ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:52.938152477 +0000 UTC m=+1921.312815655" watchObservedRunningTime="2026-07-23 18:30:52.939382365 +0000 UTC m=+1921.314045544" Jul 23 18:30:53.932030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:53.931997 2568 generic.go:356] "Generic (PLEG): container finished" podID="d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" containerID="75efec8fc2fff5e066cee2146886395dfcafe362a516ae2efa116cbea08633a8" exitCode=0 Jul 23 18:30:53.932487 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:53.932076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" event={"ID":"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54","Type":"ContainerDied","Data":"75efec8fc2fff5e066cee2146886395dfcafe362a516ae2efa116cbea08633a8"} Jul 23 18:30:55.063904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.063876 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:55.201059 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.200975 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca\") pod \"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54\" (UID: \"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54\") " Jul 23 18:30:55.201340 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.201315 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" (UID: "d4afb1ec-e120-40a2-a1c0-e7b0406f1d54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:30:55.302469 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.302432 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:30:55.942660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.942633 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" Jul 23 18:30:55.942660 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.942640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4" event={"ID":"d4afb1ec-e120-40a2-a1c0-e7b0406f1d54","Type":"ContainerDied","Data":"370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e"} Jul 23 18:30:55.942888 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:30:55.942705 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370b7206990a934b609a6a736ecd346cfea0a4e573cae84cddb522dac8ab5d0e" Jul 23 18:31:07.185203 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:07.185160 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4"] Jul 23 18:31:07.187974 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:07.187948 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5680af2-n7kp4"] Jul 23 18:31:07.323869 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:07.323832 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:31:07.324127 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:07.324082 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" containerID="cri-o://5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8" gracePeriod=30 Jul 23 18:31:08.153335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:08.153300 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" path="/var/lib/kubelet/pods/d4afb1ec-e120-40a2-a1c0-e7b0406f1d54/volumes" Jul 23 18:31:12.572529 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.572500 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:31:12.670471 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670438 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670488 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670537 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7cbh\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670593 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670643 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670675 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.670765 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670740 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.671128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.671128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670831 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo\") pod \"1b055711-6260-447a-87e4-f3c2b932560a\" (UID: \"1b055711-6260-447a-87e4-f3c2b932560a\") " Jul 23 18:31:12.671128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:31:12.671128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.670956 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:31:12.671335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671166 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:31:12.671335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671266 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.671335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671283 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.671335 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671297 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b055711-6260-447a-87e4-f3c2b932560a-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.671552 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671497 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:31:12.671944 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.671915 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:31:12.672904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.672874 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh" (OuterVolumeSpecName: "kube-api-access-q7cbh") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "kube-api-access-q7cbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:31:12.672904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.672888 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:31:12.673069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.672886 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:31:12.673069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.672976 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b055711-6260-447a-87e4-f3c2b932560a" (UID: "1b055711-6260-447a-87e4-f3c2b932560a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:31:12.772674 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772638 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.772674 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772668 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b055711-6260-447a-87e4-f3c2b932560a-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.772674 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772678 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.772674 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772699 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cbh\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-kube-api-access-q7cbh\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.772969 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772709 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b055711-6260-447a-87e4-f3c2b932560a-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:12.772969 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:12.772720 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b055711-6260-447a-87e4-f3c2b932560a-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:31:13.027274 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.027193 2568 generic.go:356] "Generic (PLEG): container finished" podID="1b055711-6260-447a-87e4-f3c2b932560a" containerID="5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8" exitCode=0 Jul 23 18:31:13.027274 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.027241 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" Jul 23 18:31:13.027457 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.027276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerDied","Data":"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8"} Jul 23 18:31:13.027457 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.027341 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp" event={"ID":"1b055711-6260-447a-87e4-f3c2b932560a","Type":"ContainerDied","Data":"be10db509938e9f6ae38c63cb94cfb8e065aab8e0d9e0f34a2842c2d34a3fe2e"} Jul 23 18:31:13.027457 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.027364 2568 scope.go:117] "RemoveContainer" containerID="5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8" Jul 23 18:31:13.042511 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.042491 2568 scope.go:117] "RemoveContainer" containerID="fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0" Jul 23 18:31:13.050214 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.050198 2568 scope.go:117] "RemoveContainer" containerID="5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8" Jul 23 18:31:13.050474 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:31:13.050454 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8\": container with ID starting with 5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8 not found: ID does not exist" containerID="5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8" Jul 23 18:31:13.050547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.050484 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8"} err="failed to get container status \"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8\": rpc error: code = NotFound desc = could not find container \"5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8\": container with ID starting with 5097944a8075ad7a990648deda84e20d68b7ea479b158d778051099d24d58cd8 not found: ID does not exist" Jul 23 18:31:13.050547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.050506 2568 scope.go:117] "RemoveContainer" containerID="fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0" Jul 23 18:31:13.050770 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:31:13.050751 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0\": container with ID starting with fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0 not found: ID does not exist" containerID="fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0" Jul 23 18:31:13.050827 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.050775 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0"} err="failed to get container status \"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0\": rpc error: code = NotFound desc = could not find container \"fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0\": container with ID starting with fcad0872426b7740fb10d4dc2d1b53c38f140f9bc80cb6b783b5f5417e032fa0 not found: ID does not exist" Jul 23 18:31:13.053919 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.053896 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:31:13.059790 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:13.059769 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b5680af2-openshift-default-747f4f4bcf-5knmp"] Jul 23 18:31:14.153647 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:31:14.153606 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b055711-6260-447a-87e4-f3c2b932560a" path="/var/lib/kubelet/pods/1b055711-6260-447a-87e4-f3c2b932560a/volumes" Jul 23 18:32:46.543557 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543522 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543879 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543891 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543903 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543908 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543929 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" containerName="revoke-keys" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543934 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" containerName="revoke-keys" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543985 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc9c7f6-82b7-4b64-a9a5-45cb0b5c880f" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.543995 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.544001 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b055711-6260-447a-87e4-f3c2b932560a" containerName="istio-proxy" Jul 23 18:32:46.544067 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.544006 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4afb1ec-e120-40a2-a1c0-e7b0406f1d54" containerName="revoke-keys" Jul 23 18:32:46.546973 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.546954 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.551714 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.551692 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e318efc2-openshift-default-dockercfg-9mxff\"" Jul 23 18:32:46.586761 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.586718 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:32:46.682107 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682107 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhms\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682465 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.682564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.682497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.783978 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.783941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.783987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784112 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784157 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhms\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784527 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784830 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.784896 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.784879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.786350 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.786328 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.786764 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.786748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.796199 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.796130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.796325 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.796226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhms\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms\") pod \"e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:46.857732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:46.857670 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:47.005106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.005079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:32:47.007281 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:32:47.007255 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb63af16_84cc_43ed_83de_a3ec06090c59.slice/crio-f7a7f2b0ba1cb6ea9a2d675aed703d9e41f5aef88162d607d8e2ff3d1d2b4922 WatchSource:0}: Error finding container f7a7f2b0ba1cb6ea9a2d675aed703d9e41f5aef88162d607d8e2ff3d1d2b4922: Status 404 returned error can't find the container with id f7a7f2b0ba1cb6ea9a2d675aed703d9e41f5aef88162d607d8e2ff3d1d2b4922 Jul 23 18:32:47.009385 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.009352 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:32:47.009488 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.009442 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:32:47.009551 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.009486 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:32:47.469170 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.469130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerStarted","Data":"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0"} Jul 23 18:32:47.469359 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.469213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerStarted","Data":"f7a7f2b0ba1cb6ea9a2d675aed703d9e41f5aef88162d607d8e2ff3d1d2b4922"} Jul 23 18:32:47.498051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.497988 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podStartSLOduration=1.497971518 podStartE2EDuration="1.497971518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:32:47.496853792 +0000 UTC m=+2035.871516969" watchObservedRunningTime="2026-07-23 18:32:47.497971518 +0000 UTC m=+2035.872634696" Jul 23 18:32:47.858655 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:47.858556 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:48.862967 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:48.862933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:49.478920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:49.478891 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:32:49.479802 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:32:49.479781 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:01.807074 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.807029 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm"] Jul 23 18:33:01.810716 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.810675 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:01.813054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.813028 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:33:01.818199 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.818162 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm"] Jul 23 18:33:01.839591 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.839553 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm\" (UID: \"5e228e4f-3801-418f-af34-e22b41ac0860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:01.940642 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.940599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm\" (UID: \"5e228e4f-3801-418f-af34-e22b41ac0860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:01.941517 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:01.941489 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm\" (UID: \"5e228e4f-3801-418f-af34-e22b41ac0860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:02.125501 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:02.125403 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:02.277336 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:02.277298 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm"] Jul 23 18:33:02.280621 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:33:02.280588 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e228e4f_3801_418f_af34_e22b41ac0860.slice/crio-fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c WatchSource:0}: Error finding container fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c: Status 404 returned error can't find the container with id fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c Jul 23 18:33:02.542355 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:02.542315 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" event={"ID":"5e228e4f-3801-418f-af34-e22b41ac0860","Type":"ContainerStarted","Data":"fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c"} Jul 23 18:33:03.549289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:03.549256 2568 generic.go:356] "Generic (PLEG): container finished" podID="5e228e4f-3801-418f-af34-e22b41ac0860" containerID="591a8c7049237d7bd032aa06102141b802f3468d5e118fd9891331f0f82753d9" exitCode=0 Jul 23 18:33:03.549645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:03.549343 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" event={"ID":"5e228e4f-3801-418f-af34-e22b41ac0860","Type":"ContainerDied","Data":"591a8c7049237d7bd032aa06102141b802f3468d5e118fd9891331f0f82753d9"} Jul 23 18:33:04.480991 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.480952 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:04.481177 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.481020 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:04.555331 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.555296 2568 generic.go:356] "Generic (PLEG): container finished" podID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerID="9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0" exitCode=0 Jul 23 18:33:04.555839 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.555376 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerDied","Data":"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0"} Jul 23 18:33:04.556362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.556012 2568 scope.go:117] "RemoveContainer" containerID="9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0" Jul 23 18:33:04.557205 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.557158 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:04.557289 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.557271 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:04.557337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.557322 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:04.721478 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.721446 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:04.769639 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.769546 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca\") pod \"5e228e4f-3801-418f-af34-e22b41ac0860\" (UID: \"5e228e4f-3801-418f-af34-e22b41ac0860\") " Jul 23 18:33:04.770000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.769973 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e228e4f-3801-418f-af34-e22b41ac0860" (UID: "5e228e4f-3801-418f-af34-e22b41ac0860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:33:04.870792 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:04.870752 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e228e4f-3801-418f-af34-e22b41ac0860-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:05.562051 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.562017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerStarted","Data":"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4"} Jul 23 18:33:05.563215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.563197 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" Jul 23 18:33:05.563318 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.563224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm" event={"ID":"5e228e4f-3801-418f-af34-e22b41ac0860","Type":"ContainerDied","Data":"fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c"} Jul 23 18:33:05.563318 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.563262 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5f0229144d8f594847e34f80da22408fb045d6ca23145a4121902fb05bdc8c" Jul 23 18:33:05.858639 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.858546 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:05.858821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.858765 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:05.858877 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:05.858814 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:06.858492 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:06.858457 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:06.858954 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:06.858517 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:07.858405 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:07.858371 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:07.858594 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:07.858432 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:08.859046 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:08.859000 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:08.859462 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:08.859064 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:09.858254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:09.858213 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 23 18:33:09.858466 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:09.858284 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 23 18:33:10.862186 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:10.862154 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:10.862585 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:10.862493 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:10.863168 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:10.863149 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:16.892966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:16.892869 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm"] Jul 23 18:33:16.894659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:16.894635 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e318efc2-f6mtm"] Jul 23 18:33:18.152776 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:18.152742 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e228e4f-3801-418f-af34-e22b41ac0860" path="/var/lib/kubelet/pods/5e228e4f-3801-418f-af34-e22b41ac0860/volumes" Jul 23 18:33:24.143096 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:24.143063 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:33:24.143538 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:24.143279 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" containerID="cri-o://690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4" gracePeriod=30 Jul 23 18:33:25.864613 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:25.864572 2568 patch_prober.go:28] interesting pod/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 23 18:33:25.865053 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:25.864652 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:33:29.396921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.396897 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:29.524571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524453 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dhms\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524732 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524833 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.524862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524863 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.525161 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524954 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:33:29.525161 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.524974 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data\") pod \"cb63af16-84cc-43ed-83de-a3ec06090c59\" (UID: \"cb63af16-84cc-43ed-83de-a3ec06090c59\") " Jul 23 18:33:29.525161 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525143 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:33:29.525315 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525195 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:33:29.525447 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525415 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.525447 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525451 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb63af16-84cc-43ed-83de-a3ec06090c59-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.525743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525464 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.525743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525636 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:33:29.525876 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.525856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:33:29.526876 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.526845 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:33:29.527039 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.527009 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:33:29.527128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.527076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:33:29.527128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.527077 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms" (OuterVolumeSpecName: "kube-api-access-6dhms") pod "cb63af16-84cc-43ed-83de-a3ec06090c59" (UID: "cb63af16-84cc-43ed-83de-a3ec06090c59"). InnerVolumeSpecName "kube-api-access-6dhms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:33:29.626346 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626308 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.626346 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626337 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.626346 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626348 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhms\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-kube-api-access-6dhms\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.626580 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626359 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.626580 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626367 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb63af16-84cc-43ed-83de-a3ec06090c59-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.626580 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.626375 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb63af16-84cc-43ed-83de-a3ec06090c59-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:33:29.688651 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.688613 2568 generic.go:356] "Generic (PLEG): container finished" podID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerID="690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4" exitCode=0 Jul 23 18:33:29.688838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.688658 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" Jul 23 18:33:29.688838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.688663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerDied","Data":"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4"} Jul 23 18:33:29.688838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.688739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549" event={"ID":"cb63af16-84cc-43ed-83de-a3ec06090c59","Type":"ContainerDied","Data":"f7a7f2b0ba1cb6ea9a2d675aed703d9e41f5aef88162d607d8e2ff3d1d2b4922"} Jul 23 18:33:29.688838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.688765 2568 scope.go:117] "RemoveContainer" containerID="690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4" Jul 23 18:33:29.697989 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.697968 2568 scope.go:117] "RemoveContainer" containerID="9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0" Jul 23 18:33:29.706121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.706103 2568 scope.go:117] "RemoveContainer" containerID="690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4" Jul 23 18:33:29.706392 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:33:29.706374 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4\": container with ID starting with 690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4 not found: ID does not exist" containerID="690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4" Jul 23 18:33:29.706439 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.706402 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4"} err="failed to get container status \"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4\": rpc error: code = NotFound desc = could not find container \"690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4\": container with ID starting with 690120341bd9a64b7a374a95f678fa8cd7720fac3cc738871676114b3f7ec3d4 not found: ID does not exist" Jul 23 18:33:29.706439 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.706420 2568 scope.go:117] "RemoveContainer" containerID="9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0" Jul 23 18:33:29.706663 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:33:29.706643 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0\": container with ID starting with 9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0 not found: ID does not exist" containerID="9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0" Jul 23 18:33:29.706724 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.706669 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0"} err="failed to get container status \"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0\": rpc error: code = NotFound desc = could not find container \"9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0\": container with ID starting with 9d4844185d8cc2ec6fc97090a31abe105aa9e64bdaae7997d1e1818b8f186ec0 not found: ID does not exist" Jul 23 18:33:29.714078 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.714053 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:33:29.722743 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:29.722715 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e318efc2-openshift-default-7d6bc87bc9-js549"] Jul 23 18:33:30.153229 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:30.153194 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" path="/var/lib/kubelet/pods/cb63af16-84cc-43ed-83de-a3ec06090c59/volumes" Jul 23 18:33:45.062123 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062087 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062484 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062496 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062508 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e228e4f-3801-418f-af34-e22b41ac0860" containerName="revoke-keys" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062514 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e228e4f-3801-418f-af34-e22b41ac0860" containerName="revoke-keys" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062568 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e228e4f-3801-418f-af34-e22b41ac0860" containerName="revoke-keys" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062578 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:33:45.062730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.062584 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:33:45.066672 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.066651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.069130 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.069111 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08efc605-openshift-default-dockercfg-7gqnf\"" Jul 23 18:33:45.082297 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.082263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:33:45.170860 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.170811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171034 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.170867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171034 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.170977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171034 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171034 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171229 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgghb\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171229 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171229 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171200 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.171366 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.171258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272071 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgghb\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272547 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272473 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272790 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272626 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272848 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.272971 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.273036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.272996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.274616 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.274587 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.274757 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.274713 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.280598 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.280567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.280823 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.280796 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgghb\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb\") pod \"e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.380296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.380257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:45.522809 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.522774 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:33:45.526569 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:33:45.526540 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4051b0bb_cbcc_4157_8b86_cf0b5d2cd0fa.slice/crio-52ce881501ca879c0a61cb10ca08214929483e6e2e1174343c262c33c089d0c5 WatchSource:0}: Error finding container 52ce881501ca879c0a61cb10ca08214929483e6e2e1174343c262c33c089d0c5: Status 404 returned error can't find the container with id 52ce881501ca879c0a61cb10ca08214929483e6e2e1174343c262c33c089d0c5 Jul 23 18:33:45.529163 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.529127 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:45.529243 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.529195 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:45.529243 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.529223 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:33:45.766864 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.766762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerStarted","Data":"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e"} Jul 23 18:33:45.766864 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.766817 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerStarted","Data":"52ce881501ca879c0a61cb10ca08214929483e6e2e1174343c262c33c089d0c5"} Jul 23 18:33:45.788851 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:45.788796 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podStartSLOduration=0.788778149 podStartE2EDuration="788.778149ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:33:45.787270699 +0000 UTC m=+2094.161933879" watchObservedRunningTime="2026-07-23 18:33:45.788778149 +0000 UTC m=+2094.163441378" Jul 23 18:33:46.381139 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:46.381099 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:46.386222 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:46.386194 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:46.771892 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:46.771807 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:33:46.772890 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:33:46.772868 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:34:02.245011 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.244896 2568 patch_prober.go:28] interesting pod/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:45270->10.132.0.48:15021: read: connection reset by peer" start-of-body= Jul 23 18:34:02.245011 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.244974 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:45270->10.132.0.48:15021: read: connection reset by peer" Jul 23 18:34:02.852384 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.852348 2568 generic.go:356] "Generic (PLEG): container finished" podID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerID="159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e" exitCode=0 Jul 23 18:34:02.852571 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.852419 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerDied","Data":"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e"} Jul 23 18:34:02.852862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.852848 2568 scope.go:117] "RemoveContainer" containerID="159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e" Jul 23 18:34:02.853838 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.853812 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:34:02.853900 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.853881 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:34:02.853935 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:02.853915 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:34:03.859852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:03.859815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerStarted","Data":"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c"} Jul 23 18:34:04.381558 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:04.381518 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:34:04.381905 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:04.381602 2568 patch_prober.go:28] interesting pod/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 23 18:34:04.382095 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:04.382070 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 23 18:34:05.381154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:05.381112 2568 patch_prober.go:28] interesting pod/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 23 18:34:05.381667 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:05.381184 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 23 18:34:06.380889 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:06.380843 2568 patch_prober.go:28] interesting pod/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 23 18:34:06.381108 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:06.380912 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 23 18:34:07.390171 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:07.390132 2568 patch_prober.go:28] interesting pod/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 23 18:34:07.390773 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:07.390739 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 23 18:34:08.384574 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:08.384534 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:34:08.384960 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:08.384942 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:34:08.385657 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:08.385638 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:34:20.711345 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.711312 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8"] Jul 23 18:34:20.711852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.711673 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:34:20.711852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.711696 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb63af16-84cc-43ed-83de-a3ec06090c59" containerName="istio-proxy" Jul 23 18:34:20.714932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.714913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:20.717183 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.717166 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\"" Jul 23 18:34:20.721296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.721274 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8"] Jul 23 18:34:20.827733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.827676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08efc605-7m9l8\" (UID: \"2c2e27cb-2634-4e10-99d1-f4a1275c8886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:20.929082 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.929031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08efc605-7m9l8\" (UID: \"2c2e27cb-2634-4e10-99d1-f4a1275c8886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:20.929706 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:20.929661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08efc605-7m9l8\" (UID: \"2c2e27cb-2634-4e10-99d1-f4a1275c8886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:21.027151 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:21.027068 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:21.156382 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:21.156358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8"] Jul 23 18:34:21.158333 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:34:21.158302 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e27cb_2634_4e10_99d1_f4a1275c8886.slice/crio-7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48 WatchSource:0}: Error finding container 7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48: Status 404 returned error can't find the container with id 7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48 Jul 23 18:34:21.957793 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:21.957759 2568 generic.go:356] "Generic (PLEG): container finished" podID="2c2e27cb-2634-4e10-99d1-f4a1275c8886" containerID="2861eb7b51bcf494f13f8883ad43b23ac5c1abf95ff37c6d6ca05ec78022473c" exitCode=0 Jul 23 18:34:21.958183 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:21.957826 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" event={"ID":"2c2e27cb-2634-4e10-99d1-f4a1275c8886","Type":"ContainerDied","Data":"2861eb7b51bcf494f13f8883ad43b23ac5c1abf95ff37c6d6ca05ec78022473c"} Jul 23 18:34:21.958183 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:21.957870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" event={"ID":"2c2e27cb-2634-4e10-99d1-f4a1275c8886","Type":"ContainerStarted","Data":"7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48"} Jul 23 18:34:23.095111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.095085 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:23.158066 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.158033 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca\") pod \"2c2e27cb-2634-4e10-99d1-f4a1275c8886\" (UID: \"2c2e27cb-2634-4e10-99d1-f4a1275c8886\") " Jul 23 18:34:23.158419 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.158395 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c2e27cb-2634-4e10-99d1-f4a1275c8886" (UID: "2c2e27cb-2634-4e10-99d1-f4a1275c8886"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:34:23.259540 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.259457 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e27cb-2634-4e10-99d1-f4a1275c8886-maas-api-service-ca\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:34:23.969618 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.969587 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" Jul 23 18:34:23.969801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.969620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8" event={"ID":"2c2e27cb-2634-4e10-99d1-f4a1275c8886","Type":"ContainerDied","Data":"7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48"} Jul 23 18:34:23.969801 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:23.969672 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a83843a29783f5b5fb69cb343c575df6bc2627bfda8b54a85c545e5c6dd4d48" Jul 23 18:34:40.790962 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:40.790930 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8"] Jul 23 18:34:40.798483 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:40.798459 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08efc605-7m9l8"] Jul 23 18:34:42.154429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:34:42.154395 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2e27cb-2634-4e10-99d1-f4a1275c8886" path="/var/lib/kubelet/pods/2c2e27cb-2634-4e10-99d1-f4a1275c8886/volumes" Jul 23 18:35:31.436367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:31.436331 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:35:31.436981 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:31.436589 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" containerID="cri-o://f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c" gracePeriod=30 Jul 23 18:35:36.682375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.682342 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:35:36.763957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.763881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgghb\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.763957 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.763935 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.763984 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764030 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764076 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764144 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764163 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert\") pod \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\" (UID: \"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa\") " Jul 23 18:35:36.764500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764270 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:36.764500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764303 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:36.764500 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764321 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:36.764638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764539 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.764638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764558 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.764638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764572 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.764638 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.764594 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:35:36.765414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.765390 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:36.766254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.766224 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:35:36.766254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.766246 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb" (OuterVolumeSpecName: "kube-api-access-jgghb") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "kube-api-access-jgghb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:35:36.766496 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.766459 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:36.766564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.766478 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" (UID: "4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:35:36.865521 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865490 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.865521 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865519 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.865521 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865531 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.865868 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865542 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.865868 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865551 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:36.865868 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:36.865563 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgghb\" (UniqueName: \"kubernetes.io/projected/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa-kube-api-access-jgghb\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:35:37.320931 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.320898 2568 generic.go:356] "Generic (PLEG): container finished" podID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerID="f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c" exitCode=0 Jul 23 18:35:37.321106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.320947 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" Jul 23 18:35:37.321106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.320964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerDied","Data":"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c"} Jul 23 18:35:37.321106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.321007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r" event={"ID":"4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa","Type":"ContainerDied","Data":"52ce881501ca879c0a61cb10ca08214929483e6e2e1174343c262c33c089d0c5"} Jul 23 18:35:37.321106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.321028 2568 scope.go:117] "RemoveContainer" containerID="f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c" Jul 23 18:35:37.330644 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.330619 2568 scope.go:117] "RemoveContainer" containerID="159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e" Jul 23 18:35:37.340540 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.340515 2568 scope.go:117] "RemoveContainer" containerID="f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c" Jul 23 18:35:37.340940 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:35:37.340875 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c\": container with ID starting with f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c not found: ID does not exist" containerID="f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c" Jul 23 18:35:37.340940 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.340911 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c"} err="failed to get container status \"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c\": rpc error: code = NotFound desc = could not find container \"f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c\": container with ID starting with f6bc1c4226392bedfb8a695bc3bde8ffd1957034cd98e60d38c56a92722bc05c not found: ID does not exist" Jul 23 18:35:37.340940 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.340934 2568 scope.go:117] "RemoveContainer" containerID="159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e" Jul 23 18:35:37.341228 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:35:37.341207 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e\": container with ID starting with 159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e not found: ID does not exist" containerID="159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e" Jul 23 18:35:37.341292 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.341232 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e"} err="failed to get container status \"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e\": rpc error: code = NotFound desc = could not find container \"159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e\": container with ID starting with 159e866a89a1b152913f539b1ca73f6a474dc9d4f236d0b38a197ab38037010e not found: ID does not exist" Jul 23 18:35:37.342993 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.342971 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:35:37.346302 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:37.346282 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08efc605-openshift-default-69b6b9d679-gkl5r"] Jul 23 18:35:38.152320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:38.152289 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" path="/var/lib/kubelet/pods/4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa/volumes" Jul 23 18:35:52.322108 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:35:52.322080 2568 scope.go:117] "RemoveContainer" containerID="d497c718b02c8f2beaa8a897905954722aa1bd4ffc6218956c9aba3651536afa" Jul 23 18:36:28.143034 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.142949 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143411 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e27cb-2634-4e10-99d1-f4a1275c8886" containerName="revoke-keys" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143431 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e27cb-2634-4e10-99d1-f4a1275c8886" containerName="revoke-keys" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143458 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143468 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143480 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143490 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143586 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2e27cb-2634-4e10-99d1-f4a1275c8886" containerName="revoke-keys" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143599 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.143617 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.143610 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4051b0bb-cbcc-4157-8b86-cf0b5d2cd0fa" containerName="istio-proxy" Jul 23 18:36:28.146876 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.146857 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.149464 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.149442 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-816eca-openshift-default-dockercfg-5b85j\"" Jul 23 18:36:28.150330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7vj\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150429 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150531 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150531 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150642 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150642 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150779 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.150779 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.150755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.159239 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.159213 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:36:28.251131 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251131 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7vj\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251386 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251559 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.251920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.251848 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.252036 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.252013 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.252129 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.252027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.253970 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.253942 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.254080 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.253989 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.264133 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.264110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.264313 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.264293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7vj\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj\") pod \"e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.458054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.457954 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:28.594530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.594508 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:36:28.596873 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:36:28.596847 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa9d7de_50d0_458d_827d_d6ce0c82db68.slice/crio-2636ecf974b099e8ff1c9497733e456979bf3fd1067a3f0f34e0ca526a1adc9a WatchSource:0}: Error finding container 2636ecf974b099e8ff1c9497733e456979bf3fd1067a3f0f34e0ca526a1adc9a: Status 404 returned error can't find the container with id 2636ecf974b099e8ff1c9497733e456979bf3fd1067a3f0f34e0ca526a1adc9a Jul 23 18:36:28.598827 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.598794 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:28.598955 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.598878 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:28.598955 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:28.598942 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:29.578701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:29.578629 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerStarted","Data":"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef"} Jul 23 18:36:29.579152 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:29.578755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerStarted","Data":"2636ecf974b099e8ff1c9497733e456979bf3fd1067a3f0f34e0ca526a1adc9a"} Jul 23 18:36:29.604163 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:29.604113 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podStartSLOduration=1.604086675 podStartE2EDuration="1.604086675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:36:29.603335436 +0000 UTC m=+2257.977998615" watchObservedRunningTime="2026-07-23 18:36:29.604086675 +0000 UTC m=+2257.978749931" Jul 23 18:36:30.458942 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:30.458902 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:30.464099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:30.464060 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:30.583610 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:30.583578 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:30.584620 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:30.584604 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:36.835107 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.835071 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:36:36.838853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.838822 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.841331 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.841309 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-629a5d-openshift-default-dockercfg-c4f74\"" Jul 23 18:36:36.849343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.849322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:36:36.930731 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.930678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931180 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931258 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zh5s\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931381 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:36.931420 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:36.931382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032656 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.032871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zh5s\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032876 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.032972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033286 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.033175 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033286 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.033201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033474 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.033449 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033569 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.033484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.033954 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.033930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.035069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.035041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.035271 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.035250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.041195 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.041167 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zh5s\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.041287 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.041251 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token\") pod \"e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.150783 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.150749 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:37.285628 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.285605 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:36:37.287591 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:36:37.287566 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf989d8b6_b3d8_4ff4_8458_ff1f395e900c.slice/crio-427884b27e14879cf35964b1f616977ab631e5f7e4278f21d0b215dece6fd3be WatchSource:0}: Error finding container 427884b27e14879cf35964b1f616977ab631e5f7e4278f21d0b215dece6fd3be: Status 404 returned error can't find the container with id 427884b27e14879cf35964b1f616977ab631e5f7e4278f21d0b215dece6fd3be Jul 23 18:36:37.289824 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.289791 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:37.289914 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.289857 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:37.289914 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.289883 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:37.618076 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.617989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerStarted","Data":"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da"} Jul 23 18:36:37.618076 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.618044 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerStarted","Data":"427884b27e14879cf35964b1f616977ab631e5f7e4278f21d0b215dece6fd3be"} Jul 23 18:36:37.639648 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:37.639587 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podStartSLOduration=1.639569785 podStartE2EDuration="1.639569785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:36:37.638112683 +0000 UTC m=+2266.012775864" watchObservedRunningTime="2026-07-23 18:36:37.639569785 +0000 UTC m=+2266.014232964" Jul 23 18:36:38.153352 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:38.153319 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:38.156317 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:38.156291 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:38.623333 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:38.623249 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:38.624201 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:38.624179 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:44.657270 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.657237 2568 generic.go:356] "Generic (PLEG): container finished" podID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerID="1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef" exitCode=0 Jul 23 18:36:44.657729 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.657283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerDied","Data":"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef"} Jul 23 18:36:44.657803 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.657787 2568 scope.go:117] "RemoveContainer" containerID="1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef" Jul 23 18:36:44.658857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.658821 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:44.658939 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.658916 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:44.658976 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:44.658963 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:45.584120 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:45.584059 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:45.670346 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:45.670308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerStarted","Data":"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e"} Jul 23 18:36:46.458262 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:46.458229 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:46.459018 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:46.458993 2568 patch_prober.go:28] interesting pod/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:36:46.459133 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:46.459054 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 23 18:36:47.458865 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:47.458816 2568 patch_prober.go:28] interesting pod/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:36:47.459295 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:47.458925 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 23 18:36:48.458749 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:48.458712 2568 patch_prober.go:28] interesting pod/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:36:48.458966 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:48.458775 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 23 18:36:49.459196 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:49.459157 2568 patch_prober.go:28] interesting pod/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:36:49.459598 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:49.459212 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 23 18:36:50.462794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:50.462763 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:50.463199 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:50.463126 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:50.463847 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:50.463831 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:36:52.344961 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:52.344930 2568 scope.go:117] "RemoveContainer" containerID="5cd852dc938fda9553f64c4c58a819ae012f745148acc0042025e8ed10952b1b" Jul 23 18:36:52.353363 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:52.353325 2568 scope.go:117] "RemoveContainer" containerID="b9c151db2a8749cead6080a55d20dfb4ccfe73f2330f1d72fd75284e8ee4be62" Jul 23 18:36:53.624780 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.624738 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:53.625166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.624811 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:36:53.719257 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.719182 2568 generic.go:356] "Generic (PLEG): container finished" podID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerID="aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da" exitCode=0 Jul 23 18:36:53.719414 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.719263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerDied","Data":"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da"} Jul 23 18:36:53.719729 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.719713 2568 scope.go:117] "RemoveContainer" containerID="aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da" Jul 23 18:36:53.720752 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.720722 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:53.720824 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.720807 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:53.720864 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:53.720847 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:36:54.725480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:54.725447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerStarted","Data":"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1"} Jul 23 18:36:55.151333 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:55.151295 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:36:55.151542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:55.151471 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:55.151542 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:55.151524 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:36:56.151223 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:56.151184 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:56.151730 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:56.151257 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:36:57.151903 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:57.151864 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:57.152281 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:57.151924 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:36:58.151909 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:58.151875 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:58.152328 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:58.151938 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:36:59.151379 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:59.151341 2568 patch_prober.go:28] interesting pod/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 23 18:36:59.151558 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:36:59.151397 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 23 18:37:00.155484 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:37:00.155453 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:37:00.155914 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:37:00.155801 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:37:00.156510 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:37:00.156492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:37:52.374481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:37:52.374403 2568 scope.go:117] "RemoveContainer" containerID="75efec8fc2fff5e066cee2146886395dfcafe362a516ae2efa116cbea08633a8" Jul 23 18:38:42.363184 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:42.363151 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:38:42.363718 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:42.363365 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" containerID="cri-o://31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e" gracePeriod=30 Jul 23 18:38:47.616974 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.616946 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:38:47.728794 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728760 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728805 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728840 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728885 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb7vj\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728943 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.728994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.728994 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.729330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729018 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.729330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729024 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:38:47.729330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket\") pod \"baa9d7de-50d0-458d-827d-d6ce0c82db68\" (UID: \"baa9d7de-50d0-458d-827d-d6ce0c82db68\") " Jul 23 18:38:47.729330 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729107 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:38:47.729506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729382 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:38:47.729506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729411 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.729506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729430 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.729506 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.729493 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:38:47.730041 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.730007 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data" (OuterVolumeSpecName: "istio-data") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:38:47.731185 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.731155 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token" (OuterVolumeSpecName: "istio-token") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:38:47.731267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.731158 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:38:47.731267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.731221 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:38:47.731382 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.731346 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj" (OuterVolumeSpecName: "kube-api-access-mb7vj") pod "baa9d7de-50d0-458d-827d-d6ce0c82db68" (UID: "baa9d7de-50d0-458d-827d-d6ce0c82db68"). InnerVolumeSpecName "kube-api-access-mb7vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:38:47.830142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830098 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830132 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830142 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830144 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7vj\" (UniqueName: \"kubernetes.io/projected/baa9d7de-50d0-458d-827d-d6ce0c82db68-kube-api-access-mb7vj\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830470 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830157 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830470 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830169 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa9d7de-50d0-458d-827d-d6ce0c82db68-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830470 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830196 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:47.830470 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:47.830206 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa9d7de-50d0-458d-827d-d6ce0c82db68-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:38:48.260589 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.260555 2568 generic.go:356] "Generic (PLEG): container finished" podID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerID="31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e" exitCode=0 Jul 23 18:38:48.260821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.260613 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" Jul 23 18:38:48.260821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.260637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerDied","Data":"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e"} Jul 23 18:38:48.260821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.260707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d" event={"ID":"baa9d7de-50d0-458d-827d-d6ce0c82db68","Type":"ContainerDied","Data":"2636ecf974b099e8ff1c9497733e456979bf3fd1067a3f0f34e0ca526a1adc9a"} Jul 23 18:38:48.260821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.260734 2568 scope.go:117] "RemoveContainer" containerID="31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e" Jul 23 18:38:48.269375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.269356 2568 scope.go:117] "RemoveContainer" containerID="1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef" Jul 23 18:38:48.277537 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.277507 2568 scope.go:117] "RemoveContainer" containerID="31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e" Jul 23 18:38:48.277902 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:38:48.277878 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e\": container with ID starting with 31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e not found: ID does not exist" containerID="31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e" Jul 23 18:38:48.277969 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.277916 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e"} err="failed to get container status \"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e\": rpc error: code = NotFound desc = could not find container \"31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e\": container with ID starting with 31dd5a17472a7bfe0bdbc71777b488baabdc38d7e2ffa8da9df2ecf5019d549e not found: ID does not exist" Jul 23 18:38:48.277969 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.277941 2568 scope.go:117] "RemoveContainer" containerID="1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef" Jul 23 18:38:48.278209 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:38:48.278195 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef\": container with ID starting with 1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef not found: ID does not exist" containerID="1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef" Jul 23 18:38:48.278261 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.278212 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef"} err="failed to get container status \"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef\": rpc error: code = NotFound desc = could not find container \"1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef\": container with ID starting with 1cea4b7a7d093b6210f3ad2c9a362c008f7308f59232fd4965f2255afb5e37ef not found: ID does not exist" Jul 23 18:38:48.306224 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.306192 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:38:48.310115 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:48.310081 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-816eca-openshift-default-597ff988f7-4bh8d"] Jul 23 18:38:50.153295 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:38:50.153262 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" path="/var/lib/kubelet/pods/baa9d7de-50d0-458d-827d-d6ce0c82db68/volumes" Jul 23 18:39:04.409139 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:04.409102 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:39:04.409673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:04.409374 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" containerID="cri-o://82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1" gracePeriod=30 Jul 23 18:39:05.338208 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.338175 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:39:05.338761 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.338738 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:05.338761 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.338760 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:05.338953 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.338877 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:05.338953 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.338891 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:05.340994 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.340974 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.344230 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.344204 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-22682a-openshift-default-dockercfg-7zvnk\"" Jul 23 18:39:05.356367 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.356338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:39:05.399052 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399310 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399310 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399389 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399428 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8x2\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399466 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.399543 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.399527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500282 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8x2\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.500760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.501166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500769 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.501166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.501166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500890 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.501166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.500957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.501337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.501214 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.502880 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.502857 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.502979 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.502965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.507893 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.507869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.508072 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.508055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8x2\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2\") pod \"e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.659955 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.659925 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:05.796782 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.796749 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:39:05.799246 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:39:05.799221 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8ce2cc_8a0c_476c_99f6_0d395f83ed32.slice/crio-7fc5541bbd66cdd7c211e5d8c777824d26bb813bb86f335f3986669dc860fc13 WatchSource:0}: Error finding container 7fc5541bbd66cdd7c211e5d8c777824d26bb813bb86f335f3986669dc860fc13: Status 404 returned error can't find the container with id 7fc5541bbd66cdd7c211e5d8c777824d26bb813bb86f335f3986669dc860fc13 Jul 23 18:39:05.801454 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.801400 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:05.801538 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.801501 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:05.801583 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:05.801567 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:06.351520 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:06.351486 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerStarted","Data":"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34"} Jul 23 18:39:06.351746 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:06.351547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerStarted","Data":"7fc5541bbd66cdd7c211e5d8c777824d26bb813bb86f335f3986669dc860fc13"} Jul 23 18:39:06.372456 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:06.372398 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podStartSLOduration=1.372380491 podStartE2EDuration="1.372380491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:06.371087458 +0000 UTC m=+2414.745750640" watchObservedRunningTime="2026-07-23 18:39:06.372380491 +0000 UTC m=+2414.747043669" Jul 23 18:39:06.660124 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:06.660083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:06.665030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:06.665007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:07.356375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:07.356347 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:07.357418 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:07.357393 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:09.674202 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.674133 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:39:09.743672 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743638 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.743889 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743708 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.743889 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.743889 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743831 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.743889 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743865 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zh5s\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.744111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743929 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.744111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.743955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.744111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744008 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.744111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket\") pod \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\" (UID: \"f989d8b6-b3d8-4ff4-8458-ff1f395e900c\") " Jul 23 18:39:09.744299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744237 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:39:09.744400 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744383 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.744482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744463 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:09.744619 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744596 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:09.744725 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744673 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:09.744927 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.744906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:09.746352 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.746327 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s" (OuterVolumeSpecName: "kube-api-access-5zh5s") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "kube-api-access-5zh5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:39:09.746553 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.746526 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:39:09.746553 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.746539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:39:09.746784 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.746758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f989d8b6-b3d8-4ff4-8458-ff1f395e900c" (UID: "f989d8b6-b3d8-4ff4-8458-ff1f395e900c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:09.845283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845248 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zh5s\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-kube-api-access-5zh5s\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845276 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845288 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845295 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845619 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845303 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845619 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845312 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845619 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845320 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:09.845619 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:09.845328 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f989d8b6-b3d8-4ff4-8458-ff1f395e900c-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:39:10.373418 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.373381 2568 generic.go:356] "Generic (PLEG): container finished" podID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerID="82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1" exitCode=0 Jul 23 18:39:10.373606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.373445 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" Jul 23 18:39:10.373606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.373449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerDied","Data":"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1"} Jul 23 18:39:10.373606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.373526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr" event={"ID":"f989d8b6-b3d8-4ff4-8458-ff1f395e900c","Type":"ContainerDied","Data":"427884b27e14879cf35964b1f616977ab631e5f7e4278f21d0b215dece6fd3be"} Jul 23 18:39:10.373606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.373578 2568 scope.go:117] "RemoveContainer" containerID="82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1" Jul 23 18:39:10.382331 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.382313 2568 scope.go:117] "RemoveContainer" containerID="aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da" Jul 23 18:39:10.390119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.390038 2568 scope.go:117] "RemoveContainer" containerID="82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1" Jul 23 18:39:10.390331 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:39:10.390310 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1\": container with ID starting with 82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1 not found: ID does not exist" containerID="82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1" Jul 23 18:39:10.390445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.390342 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1"} err="failed to get container status \"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1\": rpc error: code = NotFound desc = could not find container \"82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1\": container with ID starting with 82184e08107cf245c654829ef57e3d557fb1b09d829c348e5151797ccfd397c1 not found: ID does not exist" Jul 23 18:39:10.390445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.390366 2568 scope.go:117] "RemoveContainer" containerID="aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da" Jul 23 18:39:10.390621 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:39:10.390603 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da\": container with ID starting with aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da not found: ID does not exist" containerID="aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da" Jul 23 18:39:10.390706 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.390628 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da"} err="failed to get container status \"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da\": rpc error: code = NotFound desc = could not find container \"aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da\": container with ID starting with aea4f364e42171412d022f6949df21218f774385737ca88b54da28074d8583da not found: ID does not exist" Jul 23 18:39:10.391907 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.391889 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:39:10.398366 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:10.398346 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-629a5d-openshift-default-66c5b8959c-rhphr"] Jul 23 18:39:12.154409 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:12.154371 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" path="/var/lib/kubelet/pods/f989d8b6-b3d8-4ff4-8458-ff1f395e900c/volumes" Jul 23 18:39:13.725372 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725332 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:39:13.725920 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725903 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.726000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725925 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.726000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725944 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:13.726000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725953 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa9d7de-50d0-458d-827d-d6ce0c82db68" containerName="istio-proxy" Jul 23 18:39:13.726000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725974 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.726000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.725982 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.726256 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.726072 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.726256 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.726086 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f989d8b6-b3d8-4ff4-8458-ff1f395e900c" containerName="istio-proxy" Jul 23 18:39:13.729211 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.729183 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.731529 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.731510 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f77b55-openshift-default-dockercfg-slwwl\"" Jul 23 18:39:13.745673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.745649 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:39:13.887909 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.887867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.887921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.887978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888121 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhc5\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.888375 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.888275 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.989866 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.989866 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989844 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990111 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.989994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.990111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.990155 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990343 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.990214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhc5\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.990925 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.990894 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.991172 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.991148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.991399 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.991377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.992060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.992035 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.992439 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.992413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:13.995293 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:13.995264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:14.002227 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.002201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:14.002541 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.002516 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhc5\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:14.005060 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.005034 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy\") pod \"e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:14.047521 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.047484 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:14.219242 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.219217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:39:14.222106 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:39:14.222079 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb47047c_fedc_4117_8243_21d3ca2c6e56.slice/crio-54b3edd879e1a97ecf16a42a0d7c0bc5eb2fae1191d3e752bd30ceae7b8dc46d WatchSource:0}: Error finding container 54b3edd879e1a97ecf16a42a0d7c0bc5eb2fae1191d3e752bd30ceae7b8dc46d: Status 404 returned error can't find the container with id 54b3edd879e1a97ecf16a42a0d7c0bc5eb2fae1191d3e752bd30ceae7b8dc46d Jul 23 18:39:14.224244 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.224213 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:14.224316 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.224275 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:14.224316 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.224304 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:14.403757 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.403708 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerStarted","Data":"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12"} Jul 23 18:39:14.403952 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.403779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerStarted","Data":"54b3edd879e1a97ecf16a42a0d7c0bc5eb2fae1191d3e752bd30ceae7b8dc46d"} Jul 23 18:39:14.422982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:14.422933 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podStartSLOduration=1.422916605 podStartE2EDuration="1.422916605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:14.421809099 +0000 UTC m=+2422.796472280" watchObservedRunningTime="2026-07-23 18:39:14.422916605 +0000 UTC m=+2422.797579784" Jul 23 18:39:15.048305 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:15.048266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:15.053604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:15.053574 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:15.411382 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:15.411346 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:15.412476 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:15.412448 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:19.431394 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.431355 2568 generic.go:356] "Generic (PLEG): container finished" podID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerID="dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34" exitCode=0 Jul 23 18:39:19.431946 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.431407 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerDied","Data":"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34"} Jul 23 18:39:19.431946 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.431918 2568 scope.go:117] "RemoveContainer" containerID="dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34" Jul 23 18:39:19.433013 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.432981 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:19.433109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.433048 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:19.433109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:19.433079 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:20.438726 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:20.438672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerStarted","Data":"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4"} Jul 23 18:39:20.660435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:20.660397 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:20.661800 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:20.661774 2568 patch_prober.go:28] interesting pod/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 23 18:39:20.661932 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:20.661825 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 23 18:39:21.660593 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:21.660554 2568 patch_prober.go:28] interesting pod/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 23 18:39:21.661013 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:21.660627 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 23 18:39:22.357122 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:22.357087 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:22.661185 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:22.661157 2568 patch_prober.go:28] interesting pod/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 23 18:39:22.661556 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:22.661208 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 23 18:39:23.661255 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:23.661220 2568 patch_prober.go:28] interesting pod/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 23 18:39:23.661651 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:23.661292 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 23 18:39:24.664564 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:24.664531 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:24.665821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:24.665798 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:39:31.407467 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.407361 2568 patch_prober.go:28] interesting pod/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:43082->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 23 18:39:31.407467 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.407434 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:43082->10.132.0.53:15021: read: connection reset by peer" Jul 23 18:39:31.492917 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.492887 2568 generic.go:356] "Generic (PLEG): container finished" podID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerID="69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12" exitCode=0 Jul 23 18:39:31.493073 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.492939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerDied","Data":"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12"} Jul 23 18:39:31.493296 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.493282 2568 scope.go:117] "RemoveContainer" containerID="69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12" Jul 23 18:39:31.494214 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.494183 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:31.494276 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.494262 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:31.494327 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:31.494310 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:39:32.500340 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:32.500302 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerStarted","Data":"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813"} Jul 23 18:39:33.047745 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:33.047706 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:33.047964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:33.047938 2568 patch_prober.go:28] interesting pod/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 18:39:33.048009 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:33.047992 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 18:39:34.048845 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:34.048800 2568 patch_prober.go:28] interesting pod/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 18:39:34.049352 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:34.048872 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 18:39:35.048849 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:35.048815 2568 patch_prober.go:28] interesting pod/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 18:39:35.049027 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:35.048870 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 18:39:36.048267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:36.048229 2568 patch_prober.go:28] interesting pod/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 18:39:36.048448 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:36.048284 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 18:39:37.051473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:37.051440 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:37.051968 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:37.051840 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:37.052480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:37.052463 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:39:52.405329 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:52.405297 2568 scope.go:117] "RemoveContainer" containerID="591a8c7049237d7bd032aa06102141b802f3468d5e118fd9891331f0f82753d9" Jul 23 18:39:56.068605 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:56.068573 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:39:56.069032 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:39:56.068806 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" containerID="cri-o://92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4" gracePeriod=30 Jul 23 18:40:01.330698 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.330625 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:40:01.390699 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390659 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.390857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390728 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.390857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390751 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.390857 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390787 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.390994 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391020 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391043 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391043 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r8x2\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2\") pod \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\" (UID: \"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32\") " Jul 23 18:40:01.391304 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391064 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:01.391304 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391274 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:01.391473 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391446 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.391599 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391476 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.391599 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:01.391839 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391806 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:40:01.391995 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.391971 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:01.392901 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.392880 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:01.392998 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.392933 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:01.393042 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.393020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:40:01.393181 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.393157 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2" (OuterVolumeSpecName: "kube-api-access-9r8x2") pod "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" (UID: "8f8ce2cc-8a0c-476c-99f6-0d395f83ed32"). InnerVolumeSpecName "kube-api-access-9r8x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:01.492105 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492075 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492105 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492099 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492105 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492108 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8x2\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-kube-api-access-9r8x2\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492119 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492127 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492135 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.492337 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.492142 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:01.640002 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.639965 2568 generic.go:356] "Generic (PLEG): container finished" podID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerID="92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4" exitCode=0 Jul 23 18:40:01.640160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.640021 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" Jul 23 18:40:01.640160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.640044 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerDied","Data":"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4"} Jul 23 18:40:01.640160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.640100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv" event={"ID":"8f8ce2cc-8a0c-476c-99f6-0d395f83ed32","Type":"ContainerDied","Data":"7fc5541bbd66cdd7c211e5d8c777824d26bb813bb86f335f3986669dc860fc13"} Jul 23 18:40:01.640160 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.640121 2568 scope.go:117] "RemoveContainer" containerID="92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4" Jul 23 18:40:01.649869 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.649853 2568 scope.go:117] "RemoveContainer" containerID="dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34" Jul 23 18:40:01.657390 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.657367 2568 scope.go:117] "RemoveContainer" containerID="92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4" Jul 23 18:40:01.657614 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:40:01.657599 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4\": container with ID starting with 92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4 not found: ID does not exist" containerID="92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4" Jul 23 18:40:01.657663 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.657621 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4"} err="failed to get container status \"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4\": rpc error: code = NotFound desc = could not find container \"92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4\": container with ID starting with 92033c57bb4ba055b6b60fb39d89363717a65388ded2a3c85649b739ddd0c8e4 not found: ID does not exist" Jul 23 18:40:01.657663 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.657635 2568 scope.go:117] "RemoveContainer" containerID="dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34" Jul 23 18:40:01.657863 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:40:01.657847 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34\": container with ID starting with dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34 not found: ID does not exist" containerID="dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34" Jul 23 18:40:01.657904 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.657867 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34"} err="failed to get container status \"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34\": rpc error: code = NotFound desc = could not find container \"dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34\": container with ID starting with dc5dfa8eff7f29c69511835b7bae616145e9dad4f585906b33849a7888823c34 not found: ID does not exist" Jul 23 18:40:01.661397 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.661375 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:40:01.666959 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:01.666937 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-22682a-openshift-default-695bd79ffc-whpgv"] Jul 23 18:40:02.153161 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:02.153128 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" path="/var/lib/kubelet/pods/8f8ce2cc-8a0c-476c-99f6-0d395f83ed32/volumes" Jul 23 18:40:23.711901 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:23.711860 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:40:23.712420 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:23.712131 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" containerID="cri-o://359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813" gracePeriod=30 Jul 23 18:40:24.735797 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.735766 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:40:24.736254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736189 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.736254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736204 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.736254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736229 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.736254 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736235 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.736481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736296 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.736481 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.736308 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ce2cc-8a0c-476c-99f6-0d395f83ed32" containerName="istio-proxy" Jul 23 18:40:24.739028 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.739005 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.741551 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.741525 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5f06b1-openshift-default-dockercfg-978vz\"" Jul 23 18:40:24.752156 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.752130 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:40:24.820832 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.820795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821003 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.820857 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821003 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.820886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821003 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.820983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8x5\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.821025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.821062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.821078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.821168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.821275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.821237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.928983 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.928940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.928992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929154 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929267 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8x5\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929370 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929370 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929511 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929511 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929636 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929565 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929916 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.929916 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.929904 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.930137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.930110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.930273 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.930159 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.931751 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.931724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.931751 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.931748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.938928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.938904 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8x5\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:24.940445 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:24.940424 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token\") pod \"e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:25.050364 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.050283 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:25.190088 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.190062 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:40:25.192286 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:40:25.192258 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd77d43c_d08e_4ca0_ac11_6413f3b91bf6.slice/crio-ccf0754885985eb3b3cf84db949ac32335acad8417d8db487804cb6940c4c09a WatchSource:0}: Error finding container ccf0754885985eb3b3cf84db949ac32335acad8417d8db487804cb6940c4c09a: Status 404 returned error can't find the container with id ccf0754885985eb3b3cf84db949ac32335acad8417d8db487804cb6940c4c09a Jul 23 18:40:25.194510 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.194476 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:25.194586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.194543 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:25.194586 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.194571 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:25.760434 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.760392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerStarted","Data":"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612"} Jul 23 18:40:25.760855 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.760460 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerStarted","Data":"ccf0754885985eb3b3cf84db949ac32335acad8417d8db487804cb6940c4c09a"} Jul 23 18:40:25.784103 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:25.784048 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podStartSLOduration=1.7840316120000002 podStartE2EDuration="1.784031612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:40:25.78244347 +0000 UTC m=+2494.157106650" watchObservedRunningTime="2026-07-23 18:40:25.784031612 +0000 UTC m=+2494.158694790" Jul 23 18:40:26.051424 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:26.051337 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:26.056351 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:26.056324 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:26.765283 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:26.765247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:26.766199 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:26.766180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:28.961435 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:28.961411 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:40:29.073126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073029 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073089 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073123 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073175 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073204 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czhc5\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073330 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:29.073410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073393 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073435 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073463 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token\") pod \"db47047c-fedc-4117-8243-21d3ca2c6e56\" (UID: \"db47047c-fedc-4117-8243-21d3ca2c6e56\") " Jul 23 18:40:29.073673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073500 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:29.073673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073575 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:29.073673 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073607 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:40:29.073928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073903 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.073928 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073922 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db47047c-fedc-4117-8243-21d3ca2c6e56-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.074026 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073936 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.074026 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.073949 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.074139 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.074118 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data" (OuterVolumeSpecName: "istio-data") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:29.075436 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.075411 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:40:29.075560 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.075541 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token" (OuterVolumeSpecName: "istio-token") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:29.075698 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.075657 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:29.075840 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.075824 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5" (OuterVolumeSpecName: "kube-api-access-czhc5") pod "db47047c-fedc-4117-8243-21d3ca2c6e56" (UID: "db47047c-fedc-4117-8243-21d3ca2c6e56"). InnerVolumeSpecName "kube-api-access-czhc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:29.174581 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.174541 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhc5\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-kube-api-access-czhc5\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.174581 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.174577 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.174581 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.174587 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.174862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.174595 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.174862 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.174603 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db47047c-fedc-4117-8243-21d3ca2c6e56-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:40:29.780710 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.780661 2568 generic.go:356] "Generic (PLEG): container finished" podID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerID="359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813" exitCode=0 Jul 23 18:40:29.781080 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.780730 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" Jul 23 18:40:29.781080 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.780745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerDied","Data":"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813"} Jul 23 18:40:29.781080 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.780800 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c" event={"ID":"db47047c-fedc-4117-8243-21d3ca2c6e56","Type":"ContainerDied","Data":"54b3edd879e1a97ecf16a42a0d7c0bc5eb2fae1191d3e752bd30ceae7b8dc46d"} Jul 23 18:40:29.781080 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.780822 2568 scope.go:117] "RemoveContainer" containerID="359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813" Jul 23 18:40:29.790963 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.790889 2568 scope.go:117] "RemoveContainer" containerID="69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12" Jul 23 18:40:29.799260 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.799240 2568 scope.go:117] "RemoveContainer" containerID="359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813" Jul 23 18:40:29.799498 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:40:29.799480 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813\": container with ID starting with 359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813 not found: ID does not exist" containerID="359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813" Jul 23 18:40:29.799554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.799507 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813"} err="failed to get container status \"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813\": rpc error: code = NotFound desc = could not find container \"359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813\": container with ID starting with 359cb0e1634d73652239c53490c200149dcb5fed03da933fbd9beb57a77f1813 not found: ID does not exist" Jul 23 18:40:29.799554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.799524 2568 scope.go:117] "RemoveContainer" containerID="69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12" Jul 23 18:40:29.799784 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:40:29.799767 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12\": container with ID starting with 69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12 not found: ID does not exist" containerID="69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12" Jul 23 18:40:29.799836 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.799792 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12"} err="failed to get container status \"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12\": rpc error: code = NotFound desc = could not find container \"69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12\": container with ID starting with 69a89f7c16e69fcfee644665794eb74bea131b33130259d97a0f3ebe9353fa12 not found: ID does not exist" Jul 23 18:40:29.835733 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.835708 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:40:29.843710 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:29.843661 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f77b55-openshift-default-6648cc5989-xp76c"] Jul 23 18:40:30.159341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:30.159307 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" path="/var/lib/kubelet/pods/db47047c-fedc-4117-8243-21d3ca2c6e56/volumes" Jul 23 18:40:33.097143 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.097102 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:40:33.097597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.097460 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:40:33.097597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.097471 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:40:33.097597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.097549 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:40:33.097597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.097558 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:40:33.100747 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.100729 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.103604 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.103586 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-8ee985-openshift-default-dockercfg-rxzjb\"" Jul 23 18:40:33.110901 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.110878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111016 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.110924 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111088 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111181 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111234 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111282 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgndt\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111321 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.111363 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.111317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.115898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.115874 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:40:33.212000 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.211965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212175 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgndt\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212215 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212498 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212498 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212498 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212807 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212822 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.212921 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.212875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.213114 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.213073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.213755 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.213732 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.215307 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.215285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.215970 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.215947 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.220198 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.220173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.221131 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.221109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgndt\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt\") pod \"e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.411099 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.411050 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:33.562650 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.562626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:40:33.564707 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:40:33.564665 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103a8159_8b89_42e7_9535_112f0893a3ae.slice/crio-2e1d00b7e8dfc6a1c67ad21004c572cad698f7e4025556e6a16c6e1977dda6b6 WatchSource:0}: Error finding container 2e1d00b7e8dfc6a1c67ad21004c572cad698f7e4025556e6a16c6e1977dda6b6: Status 404 returned error can't find the container with id 2e1d00b7e8dfc6a1c67ad21004c572cad698f7e4025556e6a16c6e1977dda6b6 Jul 23 18:40:33.566767 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.566733 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:33.566885 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.566798 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:33.566885 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.566830 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:33.804244 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.804149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerStarted","Data":"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f"} Jul 23 18:40:33.804244 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.804216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerStarted","Data":"2e1d00b7e8dfc6a1c67ad21004c572cad698f7e4025556e6a16c6e1977dda6b6"} Jul 23 18:40:33.833446 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:33.833392 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podStartSLOduration=0.833373891 podStartE2EDuration="833.373891ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:40:33.832129583 +0000 UTC m=+2502.206792769" watchObservedRunningTime="2026-07-23 18:40:33.833373891 +0000 UTC m=+2502.208037072" Jul 23 18:40:34.411948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:34.411911 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:34.416872 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:34.416849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:34.808982 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:34.808904 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:34.809807 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:34.809786 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:42.806887 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:42.806848 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 18:40:42.807417 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:42.807102 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 18:40:43.862647 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.862614 2568 generic.go:356] "Generic (PLEG): container finished" podID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerID="b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612" exitCode=0 Jul 23 18:40:43.863109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.862672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerDied","Data":"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612"} Jul 23 18:40:43.863109 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.863073 2568 scope.go:117] "RemoveContainer" containerID="b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612" Jul 23 18:40:43.864070 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.864043 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:43.864137 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.864112 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:43.864174 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:43.864144 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:44.870128 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:44.870043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerStarted","Data":"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab"} Jul 23 18:40:45.051454 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:45.051408 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:45.051659 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:45.051640 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 18:40:45.051781 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:45.051726 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 18:40:46.050992 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:46.050952 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 18:40:46.051424 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:46.051017 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 18:40:47.051304 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:47.051264 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 18:40:47.051662 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:47.051326 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 18:40:48.050898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:48.050854 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 18:40:48.051126 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:48.050912 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 18:40:49.054731 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:49.054675 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:49.055164 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:49.054967 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:49.055599 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:49.055580 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:40:50.109047 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.108310 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:59488->10.132.0.55:15021: read: connection reset by peer" start-of-body= Jul 23 18:40:50.109047 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.108377 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:59488->10.132.0.55:15021: read: connection reset by peer" Jul 23 18:40:50.907094 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.907060 2568 generic.go:356] "Generic (PLEG): container finished" podID="103a8159-8b89-42e7-9535-112f0893a3ae" containerID="e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f" exitCode=0 Jul 23 18:40:50.907275 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.907133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerDied","Data":"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f"} Jul 23 18:40:50.907597 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.907583 2568 scope.go:117] "RemoveContainer" containerID="e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f" Jul 23 18:40:50.908535 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.908505 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:50.908635 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.908592 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:50.908737 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:50.908639 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 18:40:51.913883 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:51.913846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerStarted","Data":"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56"} Jul 23 18:40:52.412030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:52.411998 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:52.412219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:52.412178 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 18:40:52.412264 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:52.412228 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 18:40:52.427165 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:52.427140 2568 scope.go:117] "RemoveContainer" containerID="2861eb7b51bcf494f13f8883ad43b23ac5c1abf95ff37c6d6ca05ec78022473c" Jul 23 18:40:53.411946 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:53.411910 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 18:40:53.412359 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:53.411964 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 18:40:54.411866 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:54.411831 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 18:40:54.412094 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:54.411888 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 18:40:55.411947 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:55.411908 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 18:40:55.412131 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:55.411963 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 18:40:56.455543 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:56.455506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:56.456183 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:56.456153 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:40:56.457243 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:40:56.457215 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:48:32.774006 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:32.773923 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:48:32.774509 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:32.774149 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" containerID="cri-o://d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab" gracePeriod=30 Jul 23 18:48:34.057760 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:34.057722 2568 patch_prober.go:28] interesting pod/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:48:34.058218 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:34.057786 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:48:38.036871 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.036805 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:48:38.183732 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183700 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.183891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183784 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.183891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183838 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.183891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183871 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld8x5\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.183891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183891 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.184069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183910 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.184069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.183955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.184069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184005 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.184069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184042 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert\") pod \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\" (UID: \"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6\") " Jul 23 18:48:38.184280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184094 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:48:38.184280 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184166 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:48:38.184480 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184458 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.184554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184486 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.184554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184450 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:48:38.184554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184473 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:48:38.184884 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.184860 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:48:38.186054 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.186032 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:48:38.186152 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.186076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:48:38.186305 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.186287 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5" (OuterVolumeSpecName: "kube-api-access-ld8x5") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "kube-api-access-ld8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:48:38.186364 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.186311 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" (UID: "bd77d43c-d08e-4ca0-ac11-6413f3b91bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:48:38.195810 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.195782 2568 generic.go:356] "Generic (PLEG): container finished" podID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerID="d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab" exitCode=0 Jul 23 18:48:38.195922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.195818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerDied","Data":"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab"} Jul 23 18:48:38.195922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.195833 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" Jul 23 18:48:38.195922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.195877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7" event={"ID":"bd77d43c-d08e-4ca0-ac11-6413f3b91bf6","Type":"ContainerDied","Data":"ccf0754885985eb3b3cf84db949ac32335acad8417d8db487804cb6940c4c09a"} Jul 23 18:48:38.195922 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.195900 2568 scope.go:117] "RemoveContainer" containerID="d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab" Jul 23 18:48:38.206866 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.206849 2568 scope.go:117] "RemoveContainer" containerID="b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612" Jul 23 18:48:38.214622 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.214605 2568 scope.go:117] "RemoveContainer" containerID="d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab" Jul 23 18:48:38.214987 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:48:38.214968 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab\": container with ID starting with d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab not found: ID does not exist" containerID="d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab" Jul 23 18:48:38.215040 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.214995 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab"} err="failed to get container status \"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab\": rpc error: code = NotFound desc = could not find container \"d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab\": container with ID starting with d7edea3c8e03925a599e744e22ac1ee15203179737b123481b66f2892ba102ab not found: ID does not exist" Jul 23 18:48:38.215040 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.215011 2568 scope.go:117] "RemoveContainer" containerID="b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612" Jul 23 18:48:38.215253 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:48:38.215237 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612\": container with ID starting with b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612 not found: ID does not exist" containerID="b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612" Jul 23 18:48:38.215299 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.215256 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612"} err="failed to get container status \"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612\": rpc error: code = NotFound desc = could not find container \"b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612\": container with ID starting with b4dee9c5b459712ba470ffde1622f70a85c5ef5b2df7e68c44b6a3dc69087612 not found: ID does not exist" Jul 23 18:48:38.221200 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.221181 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:48:38.225968 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.225947 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5f06b1-openshift-default-6bd8f45795-pmwd7"] Jul 23 18:48:38.285891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285862 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8x5\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-kube-api-access-ld8x5\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.285891 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285887 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.286030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285898 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.286030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285906 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.286030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285914 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.286030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285922 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:38.286030 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:38.285929 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:48:40.153290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:40.153255 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" path="/var/lib/kubelet/pods/bd77d43c-d08e-4ca0-ac11-6413f3b91bf6/volumes" Jul 23 18:48:55.659525 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:55.659487 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:48:55.660020 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:55.659774 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" containerID="cri-o://44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56" gracePeriod=30 Jul 23 18:48:56.458293 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:56.458251 2568 patch_prober.go:28] interesting pod/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:48:56.458468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:48:56.458320 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:49:00.915070 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.915040 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:49:00.986919 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.986844 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.986919 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.986914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.986956 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.986972 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987001 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgndt\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987054 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987108 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987141 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987135 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket\") pod \"103a8159-8b89-42e7-9535-112f0893a3ae\" (UID: \"103a8159-8b89-42e7-9535-112f0893a3ae\") " Jul 23 18:49:00.987554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987266 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:49:00.987554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987454 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:49:00.987554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987478 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-certs\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:00.987554 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987480 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:49:00.987783 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987643 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:49:00.987843 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.987795 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:49:00.989029 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.989005 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:49:00.989223 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.989203 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:49:00.989223 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.989211 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:49:00.989468 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:00.989449 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt" (OuterVolumeSpecName: "kube-api-access-xgndt") pod "103a8159-8b89-42e7-9535-112f0893a3ae" (UID: "103a8159-8b89-42e7-9535-112f0893a3ae"). InnerVolumeSpecName "kube-api-access-xgndt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:49:01.088119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088089 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103a8159-8b89-42e7-9535-112f0893a3ae-istiod-ca-cert\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088117 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgndt\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-kube-api-access-xgndt\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088119 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088127 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103a8159-8b89-42e7-9535-112f0893a3ae-istio-token\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088136 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-credential-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088144 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-workload-socket\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088153 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-envoy\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088161 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103a8159-8b89-42e7-9535-112f0893a3ae-istio-podinfo\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.088341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.088169 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103a8159-8b89-42e7-9535-112f0893a3ae-istio-data\") on node \"ip-10-0-129-67.ec2.internal\" DevicePath \"\"" Jul 23 18:49:01.306035 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.305941 2568 generic.go:356] "Generic (PLEG): container finished" podID="103a8159-8b89-42e7-9535-112f0893a3ae" containerID="44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56" exitCode=0 Jul 23 18:49:01.306035 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.305990 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" Jul 23 18:49:01.306226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.306022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerDied","Data":"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56"} Jul 23 18:49:01.306226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.306083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh" event={"ID":"103a8159-8b89-42e7-9535-112f0893a3ae","Type":"ContainerDied","Data":"2e1d00b7e8dfc6a1c67ad21004c572cad698f7e4025556e6a16c6e1977dda6b6"} Jul 23 18:49:01.306226 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.306105 2568 scope.go:117] "RemoveContainer" containerID="44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56" Jul 23 18:49:01.315410 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.315391 2568 scope.go:117] "RemoveContainer" containerID="e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f" Jul 23 18:49:01.322905 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.322886 2568 scope.go:117] "RemoveContainer" containerID="44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56" Jul 23 18:49:01.323154 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:49:01.323135 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56\": container with ID starting with 44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56 not found: ID does not exist" containerID="44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56" Jul 23 18:49:01.323195 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.323163 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56"} err="failed to get container status \"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56\": rpc error: code = NotFound desc = could not find container \"44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56\": container with ID starting with 44f6cdee9c35f3bec5e9f2024ecdc9304455b4d668916b5f8138c3dc63fcba56 not found: ID does not exist" Jul 23 18:49:01.323195 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.323181 2568 scope.go:117] "RemoveContainer" containerID="e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f" Jul 23 18:49:01.323404 ip-10-0-129-67 kubenswrapper[2568]: E0723 18:49:01.323386 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f\": container with ID starting with e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f not found: ID does not exist" containerID="e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f" Jul 23 18:49:01.323450 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.323410 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f"} err="failed to get container status \"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f\": rpc error: code = NotFound desc = could not find container \"e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f\": container with ID starting with e178c3f78bd2e7474b0a121b3b13d70a610777bcf3468ba8ac4719c00084ca4f not found: ID does not exist" Jul 23 18:49:01.330178 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.330156 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:49:01.334317 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:01.334296 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-8ee985-openshift-default-667d9c59c9-v4xqh"] Jul 23 18:49:02.154106 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:02.154072 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" path="/var/lib/kubelet/pods/103a8159-8b89-42e7-9535-112f0893a3ae/volumes" Jul 23 18:49:02.950836 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:02.950804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-fzzmf_4c8926d5-8943-47e6-a83b-d18eb092d0db/copy-manifests/0.log" Jul 23 18:49:02.958593 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:02.958562 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-fzzmf_4c8926d5-8943-47e6-a83b-d18eb092d0db/manager/0.log" Jul 23 18:49:03.617970 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:03.617940 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-684cc7d4cf-d6jnx_448b23cd-aa05-407b-97b8-915e2be17994/maas-api/0.log" Jul 23 18:49:03.878006 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:03.877924 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747190-c62fw_4814fd81-a7ca-49c1-9c72-1a88284a185a/cleanup/0.log" Jul 23 18:49:06.813380 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:06.813350 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f2m5l_59026f15-d08a-4c2b-9f89-0e68e37ae323/istio-proxy/0.log" Jul 23 18:49:06.940944 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:06.940917 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zl92g_0c1c4126-6271-4f5e-b2c7-10a22b234098/discovery/0.log" Jul 23 18:49:07.313069 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:07.313042 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/1.log" Jul 23 18:49:08.058021 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:08.057994 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t8sc4_15cfff98-e7e2-4e0f-ba45-23f5d358eda8/main/0.log" Jul 23 18:49:08.430204 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:08.430175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4j6kk_305daefc-12e3-4020-9f71-4dfdc1b15898/main/0.log" Jul 23 18:49:17.820292 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:17.820248 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qv8b5_1feee572-8d94-45fe-8be3-b7348a5bfa66/global-pull-secret-syncer/0.log" Jul 23 18:49:17.967134 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:17.967104 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jswkv_9447ea52-6b76-4675-a2fd-447b95a0b740/konnectivity-agent/0.log" Jul 23 18:49:18.021835 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:18.021794 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-67.ec2.internal_100d0bec6eaa8166829fa6f437da8a36/haproxy/0.log" Jul 23 18:49:25.073577 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.073551 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/alertmanager/0.log" Jul 23 18:49:25.102310 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.102282 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/config-reloader/0.log" Jul 23 18:49:25.140894 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.140872 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/kube-rbac-proxy-web/0.log" Jul 23 18:49:25.168291 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.168269 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/kube-rbac-proxy/0.log" Jul 23 18:49:25.191775 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.191749 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/kube-rbac-proxy-metric/0.log" Jul 23 18:49:25.214501 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.214478 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/prom-label-proxy/0.log" Jul 23 18:49:25.243161 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.243096 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a125d2fd-5c7b-44a9-bd64-53c2ec80a195/init-config-reloader/0.log" Jul 23 18:49:25.289953 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.289930 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-flk6k_febde259-4b55-421f-a4ec-1b4b78af5eac/cluster-monitoring-operator/0.log" Jul 23 18:49:25.537351 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.537264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fb598c4fd-z5xht_be25b4be-5347-41ed-8205-f1bc60e0ee7e/metrics-server/0.log" Jul 23 18:49:25.742320 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.742288 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4zgm_d777d537-fec3-4930-8c4f-697074fdc4b6/node-exporter/0.log" Jul 23 18:49:25.764937 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.764913 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4zgm_d777d537-fec3-4930-8c4f-697074fdc4b6/kube-rbac-proxy/0.log" Jul 23 18:49:25.785933 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.785907 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4zgm_d777d537-fec3-4930-8c4f-697074fdc4b6/init-textfile/0.log" Jul 23 18:49:25.979893 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:25.979867 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/prometheus/0.log" Jul 23 18:49:26.008485 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.008458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/config-reloader/0.log" Jul 23 18:49:26.035371 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.035347 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/thanos-sidecar/0.log" Jul 23 18:49:26.060365 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.060325 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/kube-rbac-proxy-web/0.log" Jul 23 18:49:26.084253 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.084224 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/kube-rbac-proxy/0.log" Jul 23 18:49:26.107252 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.107225 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/kube-rbac-proxy-thanos/0.log" Jul 23 18:49:26.135821 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.135796 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7e09b363-b25f-42d0-b01b-615f2ceab43c/init-config-reloader/0.log" Jul 23 18:49:26.220514 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:26.220431 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-6qncf_af882abd-0c56-4f18-bd04-adfa89e66794/prometheus-operator-admission-webhook/0.log" Jul 23 18:49:27.445149 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445115 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4"] Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445453 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445464 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445473 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445479 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="db47047c-fedc-4117-8243-21d3ca2c6e56" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445493 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445499 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445510 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.445530 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445515 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.445853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445572 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" Jul 23 18:49:27.445853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445580 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.445853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445588 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="103a8159-8b89-42e7-9535-112f0893a3ae" containerName="istio-proxy" Jul 23 18:49:27.445853 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.445594 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd77d43c-d08e-4ca0-ac11-6413f3b91bf6" containerName="istio-proxy" Jul 23 18:49:27.448778 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.448758 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.451251 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.451232 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpdkp\"/\"openshift-service-ca.crt\"" Jul 23 18:49:27.452166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.452138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cpdkp\"/\"default-dockercfg-m9lj7\"" Jul 23 18:49:27.452166 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.452147 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpdkp\"/\"kube-root-ca.crt\"" Jul 23 18:49:27.459606 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.459581 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4"] Jul 23 18:49:27.552072 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.552041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-podres\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.552252 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.552086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-lib-modules\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.552252 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.552164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-sys\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.552252 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.552229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm7h\" (UniqueName: \"kubernetes.io/projected/82d7affd-cf89-4360-8235-8424afe204bf-kube-api-access-pxm7h\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.552362 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.552283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-proc\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.652948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.652909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-sys\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.652948 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.652952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm7h\" (UniqueName: \"kubernetes.io/projected/82d7affd-cf89-4360-8235-8424afe204bf-kube-api-access-pxm7h\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.652991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-proc\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-podres\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-sys\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-lib-modules\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-proc\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653219 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653193 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-lib-modules\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.653424 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.653293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82d7affd-cf89-4360-8235-8424afe204bf-podres\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.661482 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.661455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm7h\" (UniqueName: \"kubernetes.io/projected/82d7affd-cf89-4360-8235-8424afe204bf-kube-api-access-pxm7h\") pod \"perf-node-gather-daemonset-m6cl4\" (UID: \"82d7affd-cf89-4360-8235-8424afe204bf\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.758983 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.758890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:27.889639 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:27.889611 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4"] Jul 23 18:49:27.892072 ip-10-0-129-67 kubenswrapper[2568]: W0723 18:49:27.892045 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82d7affd_cf89_4360_8235_8424afe204bf.slice/crio-91654e1ffc712a889303f13b6940ec02c569af00fae9ebbcdb5e59ca157e6c41 WatchSource:0}: Error finding container 91654e1ffc712a889303f13b6940ec02c569af00fae9ebbcdb5e59ca157e6c41: Status 404 returned error can't find the container with id 91654e1ffc712a889303f13b6940ec02c569af00fae9ebbcdb5e59ca157e6c41 Jul 23 18:49:28.428288 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.428246 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" event={"ID":"82d7affd-cf89-4360-8235-8424afe204bf","Type":"ContainerStarted","Data":"740a843890c27ad2adb462f94622b917095c161c8f569fa7e0e3c8314ac9f0e2"} Jul 23 18:49:28.428499 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.428300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" event={"ID":"82d7affd-cf89-4360-8235-8424afe204bf","Type":"ContainerStarted","Data":"91654e1ffc712a889303f13b6940ec02c569af00fae9ebbcdb5e59ca157e6c41"} Jul 23 18:49:28.428499 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.428361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:28.447262 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.447214 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" podStartSLOduration=1.44720018 podStartE2EDuration="1.44720018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:28.445576717 +0000 UTC m=+3036.820239898" watchObservedRunningTime="2026-07-23 18:49:28.44720018 +0000 UTC m=+3036.821863358" Jul 23 18:49:28.905898 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.905871 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64646bd657-pnzs5_e19b9d3d-9f18-4ec2-b59c-6adecc3ec995/console/0.log" Jul 23 18:49:28.939734 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:28.939673 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-lx2mz_06d151cf-b03b-45af-a6bb-beaa21499166/download-server/0.log" Jul 23 18:49:30.817964 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:30.817935 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j7q2q_f2e796a2-1a03-48bf-85b6-e8a1f549b5fd/dns/0.log" Jul 23 18:49:30.840147 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:30.840122 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j7q2q_f2e796a2-1a03-48bf-85b6-e8a1f549b5fd/kube-rbac-proxy/0.log" Jul 23 18:49:30.893459 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:30.893429 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bvbtd_ef4c5948-06e9-4626-8545-e242dde01132/dns-node-resolver/0.log" Jul 23 18:49:31.533100 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:31.533066 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjkxn_ae691532-a868-4769-bde5-c93f7eb4e1d0/node-ca/0.log" Jul 23 18:49:32.557326 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:32.557297 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f2m5l_59026f15-d08a-4c2b-9f89-0e68e37ae323/istio-proxy/0.log" Jul 23 18:49:32.798484 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:32.798449 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zl92g_0c1c4126-6271-4f5e-b2c7-10a22b234098/discovery/0.log" Jul 23 18:49:32.886472 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:32.886440 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/0.log" Jul 23 18:49:32.913027 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:32.912996 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/1.log" Jul 23 18:49:33.745165 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:33.745135 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnj5r_0437e524-41b9-4b65-a9aa-6f24c560e36d/serve-healthcheck-canary/0.log" Jul 23 18:49:34.371701 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:34.371660 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxkrb_43f5fa97-3a75-482a-a435-5dff531f81e0/kube-rbac-proxy/0.log" Jul 23 18:49:34.393767 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:34.393738 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxkrb_43f5fa97-3a75-482a-a435-5dff531f81e0/exporter/0.log" Jul 23 18:49:34.415591 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:34.415557 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxkrb_43f5fa97-3a75-482a-a435-5dff531f81e0/extractor/0.log" Jul 23 18:49:34.443645 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:34.443615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-m6cl4" Jul 23 18:49:37.375537 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:37.375507 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-fzzmf_4c8926d5-8943-47e6-a83b-d18eb092d0db/manager/0.log" Jul 23 18:49:37.404954 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:37.404878 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-fzzmf_4c8926d5-8943-47e6-a83b-d18eb092d0db/copy-manifests/0.log" Jul 23 18:49:38.258852 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:38.258822 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f2m5l_59026f15-d08a-4c2b-9f89-0e68e37ae323/istio-proxy/0.log" Jul 23 18:49:38.510727 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:38.510698 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zl92g_0c1c4126-6271-4f5e-b2c7-10a22b234098/discovery/0.log" Jul 23 18:49:38.571271 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:38.571238 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/0.log" Jul 23 18:49:38.594992 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:38.594962 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/1.log" Jul 23 18:49:39.028187 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:39.028107 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7ff5b44f4d-2rp6p_5245db9c-8eb0-49d6-8e24-35b8d9b79c12/manager/0.log" Jul 23 18:49:39.826836 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:39.826791 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f2m5l_59026f15-d08a-4c2b-9f89-0e68e37ae323/istio-proxy/0.log" Jul 23 18:49:40.074636 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:40.074559 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zl92g_0c1c4126-6271-4f5e-b2c7-10a22b234098/discovery/0.log" Jul 23 18:49:40.138341 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:40.138311 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/0.log" Jul 23 18:49:40.166448 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:40.166419 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jl8j8_74ab0959-51b3-4f2e-92f7-1dbea9160bd8/istio-proxy/1.log" Jul 23 18:49:45.592631 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.592601 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8bbvd_8436bc20-fc7b-4845-b5df-7b681ee5fee0/kube-multus/0.log" Jul 23 18:49:45.782070 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.782041 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/kube-multus-additional-cni-plugins/0.log" Jul 23 18:49:45.805976 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.805954 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/egress-router-binary-copy/0.log" Jul 23 18:49:45.829384 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.829358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/cni-plugins/0.log" Jul 23 18:49:45.853288 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.853214 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/bond-cni-plugin/0.log" Jul 23 18:49:45.875415 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.875394 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/routeoverride-cni/0.log" Jul 23 18:49:45.897774 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.897752 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/whereabouts-cni-bincopy/0.log" Jul 23 18:49:45.922290 ip-10-0-129-67 kubenswrapper[2568]: I0723 18:49:45.922264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gglm_d03e58df-c1b1-4de8-916d-b68257dd562e/whereabouts-cni/0.log"