Jul 30 03:45:18.572479 ip-10-0-134-114 systemd[1]: Starting Kubernetes Kubelet... Jul 30 03:45:19.004669 ip-10-0-134-114 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 03:45:19.004669 ip-10-0-134-114 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 03:45:19.004669 ip-10-0-134-114 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 03:45:19.004669 ip-10-0-134-114 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 03:45:19.004669 ip-10-0-134-114 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 03:45:19.006205 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.006100 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 03:45:19.009680 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009664 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009682 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009686 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009690 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009693 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009696 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009700 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009704 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009707 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009710 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009712 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009715 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009718 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009721 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009724 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009727 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.009721 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009730 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009732 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009735 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009738 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009743 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009746 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009748 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009751 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009753 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009756 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009758 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009760 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009763 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009765 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009768 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009770 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009773 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009775 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009778 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009781 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.010116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009783 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009785 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009788 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009790 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009792 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009795 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009797 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009799 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009802 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009805 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009809 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009813 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009815 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009818 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009821 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009824 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009827 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009829 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009832 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.010646 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009835 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009837 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009840 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009842 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009844 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009847 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009850 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009853 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009856 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009858 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009862 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009864 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009867 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009869 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009872 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009874 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009877 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009879 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009882 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009885 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.011116 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009887 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009890 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009893 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009895 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009898 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009900 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009902 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009907 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009909 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009912 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009915 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009917 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009919 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009922 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009924 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009927 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009930 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009932 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009935 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009937 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.011616 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009940 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009942 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009945 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.009947 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010680 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010687 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010690 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010693 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010697 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010700 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010703 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010706 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010709 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010711 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010714 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010717 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010720 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010723 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010727 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010731 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.012095 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010734 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010737 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010741 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010743 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010746 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010749 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010751 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010754 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010756 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010759 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010761 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010764 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010767 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010770 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010772 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010775 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010777 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010780 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010782 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010785 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.012636 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010787 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010791 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010793 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010795 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010798 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010801 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010803 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010805 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010809 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010811 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010813 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010817 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010819 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010821 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010824 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010826 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010828 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010831 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010833 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010836 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.013149 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010838 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010841 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010843 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010846 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010849 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010852 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010854 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010857 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010860 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010862 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010864 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010867 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010869 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010872 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010874 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010876 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010879 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010881 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010883 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010886 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.013647 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010888 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010892 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010894 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010897 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010901 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010903 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010906 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010910 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010913 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010916 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010919 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010922 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010925 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010928 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010931 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010933 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010936 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010939 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010942 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010945 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.014135 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010948 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010951 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.010954 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011352 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011359 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011363 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011367 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011369 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011372 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011375 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011378 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011381 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011383 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011392 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011395 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011398 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011401 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011403 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011406 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.014656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011409 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011412 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011415 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011418 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011421 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011423 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011426 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011429 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011432 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011435 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011437 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011440 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011443 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011445 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011448 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011450 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011453 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011456 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011458 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.015137 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011461 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011463 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011466 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011468 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011471 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011473 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011475 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011478 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011485 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011491 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011494 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011496 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011499 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011501 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011504 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011506 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011509 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011511 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011514 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011516 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.015706 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011521 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011523 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011527 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011529 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011531 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011534 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011536 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011539 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011541 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011544 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011546 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011548 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011551 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011553 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011556 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011559 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011561 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011564 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011566 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011568 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.016202 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011571 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011580 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011583 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011585 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011588 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011590 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011592 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011595 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011599 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011602 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011605 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011608 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011612 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011614 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011617 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011619 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011622 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011624 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011627 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.016703 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011630 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011632 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011635 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011637 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.011639 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011708 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011721 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011727 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011731 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011735 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011738 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011742 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011746 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011750 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011755 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011759 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011762 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011766 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011768 2568 flags.go:64] FLAG: --cgroup-root="" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011771 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011774 2568 flags.go:64] FLAG: --client-ca-file="" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011777 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011780 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 03:45:19.017182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011784 2568 flags.go:64] FLAG: --cluster-domain="" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011787 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011791 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011794 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011797 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011801 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011804 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011806 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011809 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011813 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011816 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011819 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011822 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011824 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011829 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011832 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011834 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011837 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011841 2568 flags.go:64] FLAG: --enable-server="true" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011844 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011848 2568 flags.go:64] FLAG: --event-burst="100" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011851 2568 flags.go:64] FLAG: --event-qps="50" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011854 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011858 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 03:45:19.017728 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011861 2568 flags.go:64] FLAG: --eviction-hard="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011865 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011868 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011886 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011890 2568 flags.go:64] FLAG: --eviction-soft="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011893 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011897 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011900 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011902 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011905 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011910 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011913 2568 flags.go:64] FLAG: --feature-gates="" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011917 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011920 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011923 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011926 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011929 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011932 2568 flags.go:64] FLAG: --help="false" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011934 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011937 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011940 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011943 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011947 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 03:45:19.018357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011950 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011953 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011955 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011958 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011961 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011963 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011966 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011969 2568 flags.go:64] FLAG: --kube-reserved="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011974 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011977 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011979 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011983 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011986 2568 flags.go:64] FLAG: --lock-file="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011989 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011992 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.011994 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012000 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012002 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012005 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012009 2568 flags.go:64] FLAG: --logging-format="text" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012012 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012015 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012018 2568 flags.go:64] FLAG: --manifest-url="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012021 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012025 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 03:45:19.018986 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012028 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012032 2568 flags.go:64] FLAG: --max-pods="110" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012035 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012038 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012041 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012044 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012046 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012049 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012052 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012060 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012062 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012066 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012069 2568 flags.go:64] FLAG: --pod-cidr="" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012071 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012076 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012080 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012083 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012087 2568 flags.go:64] FLAG: --port="10250" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012089 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012093 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d4cc31b74244eae" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012097 2568 flags.go:64] FLAG: --qos-reserved="" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012100 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012103 2568 flags.go:64] FLAG: --register-node="true" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012105 2568 flags.go:64] FLAG: --register-with-taints="" Jul 30 03:45:19.019628 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012109 2568 flags.go:64] FLAG: --registry-burst="10" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012112 2568 flags.go:64] FLAG: --registry-qps="5" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012115 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012118 2568 flags.go:64] FLAG: --reserved-memory="" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012122 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012124 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012127 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012130 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012133 2568 flags.go:64] FLAG: --runonce="false" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012135 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012138 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012141 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012144 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012147 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012149 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012152 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012155 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012158 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012161 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012179 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012183 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012188 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012191 2568 flags.go:64] FLAG: --system-cgroups="" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012195 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012201 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 03:45:19.020253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012204 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012207 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012212 2568 flags.go:64] FLAG: --tls-min-version="" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012215 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012218 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012221 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012224 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012227 2568 flags.go:64] FLAG: --v="2" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012231 2568 flags.go:64] FLAG: --version="false" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012235 2568 flags.go:64] FLAG: --vmodule="" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012239 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.012242 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012371 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012377 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012380 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012383 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012386 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012389 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012392 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012394 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012396 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012399 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.020880 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012402 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012405 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012409 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012412 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012414 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012417 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012419 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012422 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012426 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012428 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012431 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012434 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012436 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012439 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012442 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012444 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012447 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012449 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012452 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012454 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.021441 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012457 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012460 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012462 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012465 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012467 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012470 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012472 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012474 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012478 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012481 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012484 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012486 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012489 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012491 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012494 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012496 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012499 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012501 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012503 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012506 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.021977 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012510 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012512 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012515 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012517 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012520 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012522 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012525 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012527 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012530 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012532 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012535 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012537 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012540 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012542 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012544 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012547 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012549 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012552 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012554 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012556 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.022497 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012559 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012561 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012564 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012566 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012568 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012571 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012574 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012578 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012581 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012583 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012586 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012588 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012592 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012594 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012597 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012600 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012603 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012606 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012608 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.022987 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012611 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012614 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012616 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012619 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012621 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012624 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012626 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012628 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012631 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.012633 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.013322 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.020423 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.020438 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.020454 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.020463 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 03:45:19.023474 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020518 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020524 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020527 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020531 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020534 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020536 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020540 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020542 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020545 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020548 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020551 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020555 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020559 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020562 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020567 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020571 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020574 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020577 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020581 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.023857 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020583 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020586 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020588 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020591 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020593 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020596 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020599 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020601 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020604 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020606 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020608 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020611 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020614 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020618 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020621 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020624 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020627 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020630 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020633 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.024344 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020636 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020639 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020643 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020645 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020657 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020660 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020663 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020666 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020668 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020671 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020674 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020676 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020679 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020684 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020686 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020689 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020692 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020695 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020697 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020700 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.024822 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020702 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020705 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020708 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020710 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020713 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020715 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020718 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020720 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020723 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020725 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020728 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020731 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020734 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020736 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020738 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020741 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020744 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020746 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020749 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020751 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.025337 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020754 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020756 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020759 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020762 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020764 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020766 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020769 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020772 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020775 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020777 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020780 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020782 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020785 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020787 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020789 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020792 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020795 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020797 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020800 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020802 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.025835 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020804 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.020809 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020926 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020931 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020933 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020937 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020940 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020942 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020946 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020950 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020953 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020956 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020959 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020962 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020964 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 03:45:19.026338 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020967 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020970 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020973 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020975 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020979 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020981 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020983 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020986 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020988 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020991 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020993 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020995 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.020998 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021000 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021003 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021005 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021008 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021010 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021012 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021015 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 03:45:19.026711 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021017 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021020 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021022 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021024 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021027 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021030 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021032 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021035 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021037 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021040 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021042 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021045 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021047 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021049 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021052 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021054 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021057 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021060 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021062 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021065 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 03:45:19.027211 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021067 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021069 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021072 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021074 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021077 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021080 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021083 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021086 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021088 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021090 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021093 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021095 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021098 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021100 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021103 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021105 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021108 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021112 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021114 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021117 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 03:45:19.027719 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021119 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021122 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021124 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021127 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021129 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021131 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021134 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021137 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021139 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021143 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021146 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021149 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021152 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021155 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021158 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021160 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021162 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021178 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021180 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 03:45:19.028277 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021183 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021185 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021188 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021192 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021194 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021197 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:19.021200 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.021205 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.021314 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.023391 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.024983 2568 server.go:1019] "Starting client certificate rotation" Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.025094 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.025134 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 03:45:19.028958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.027618 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 03:45:19.031709 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.031682 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 03:45:19.043231 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.043205 2568 log.go:25] "Validated CRI v1 runtime API" Jul 30 03:45:19.045518 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.045500 2568 log.go:25] "Validated CRI v1 image API" Jul 30 03:45:19.046402 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.046389 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 03:45:19.050246 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.050226 2568 fs.go:135] Filesystem UUIDs: map[6d8b26e3-8d19-4541-ac5d-6946e9bb75e4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b2b8cb19-ae97-467a-bd51-ebcbdf0684a2:/dev/nvme0n1p4] Jul 30 03:45:19.050312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.050246 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 03:45:19.056072 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.055922 2568 manager.go:216] Machine: {Timestamp:2026-07-30 03:45:19.054364384 +0000 UTC m=+0.357161097 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092709 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec232b0dabf56179bc1553a87a46cd5a SystemUUID:ec232b0d-abf5-6179-bc15-53a87a46cd5a BootID:de668137-2c50-41b9-bc30-9ccf3291efe8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:6b:38:ad:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:6b:38:ad:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:cb:53:5c:0a:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 03:45:19.056181 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.056073 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 03:45:19.056224 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.056200 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 03:45:19.058014 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.057997 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 03:45:19.059650 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.059620 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 03:45:19.059807 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.059654 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-114.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 03:45:19.060411 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.060401 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 03:45:19.060451 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.060414 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 03:45:19.060451 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.060422 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 03:45:19.060451 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.060437 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 03:45:19.061213 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.061203 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 03:45:19.063033 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063021 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 30 03:45:19.063159 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063149 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 03:45:19.063212 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063197 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 30 03:45:19.063212 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063208 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 03:45:19.063291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063223 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 03:45:19.063291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063232 2568 kubelet.go:398] "Adding apiserver pod source" Jul 30 03:45:19.063291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063243 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 03:45:19.063616 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063603 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 03:45:19.063654 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.063648 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 03:45:19.066768 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.066750 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 03:45:19.069516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.069493 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 03:45:19.069611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.069544 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 03:45:19.071666 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071649 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 03:45:19.071666 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071669 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071675 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071681 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071690 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071695 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071704 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071709 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071716 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071722 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071734 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 03:45:19.071767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.071739 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 03:45:19.072584 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.072574 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 03:45:19.072621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.072584 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 03:45:19.073855 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.073827 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 03:45:19.073921 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.073854 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 03:45:19.075822 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.075808 2568 server.go:1268] "Started kubelet" Jul 30 03:45:19.075931 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.075902 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 03:45:19.075999 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.075912 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 03:45:19.076063 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.076029 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 03:45:19.076847 ip-10-0-134-114 systemd[1]: Started Kubernetes Kubelet. Jul 30 03:45:19.077560 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.077537 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 03:45:19.078137 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.078126 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 30 03:45:19.081521 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.081502 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 03:45:19.084406 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.084385 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 03:45:19.084511 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.084391 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 03:45:19.084942 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.084599 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.084942 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.084869 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vz2c6" Jul 30 03:45:19.085075 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085054 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 03:45:19.085075 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085054 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 03:45:19.085075 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085076 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 03:45:19.085260 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085243 2568 factory.go:55] Registering systemd factory Jul 30 03:45:19.085342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085265 2568 factory.go:223] Registration of the systemd container factory successfully Jul 30 03:45:19.085342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085301 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 30 03:45:19.085342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085309 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 30 03:45:19.085472 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085382 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-114.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 03:45:19.085617 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085600 2568 factory.go:153] Registering CRI-O factory Jul 30 03:45:19.085617 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085619 2568 factory.go:223] Registration of the crio container factory successfully Jul 30 03:45:19.085731 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085669 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 03:45:19.085731 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085685 2568 factory.go:103] Registering Raw factory Jul 30 03:45:19.085731 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.085695 2568 manager.go:1196] Started watching for new ooms in manager Jul 30 03:45:19.086066 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.086049 2568 manager.go:349] Starting recovery of all containers Jul 30 03:45:19.086423 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.085415 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-114.ec2.internal.18c6f47cc87dab30 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-114.ec2.internal,UID:ip-10-0-134-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-114.ec2.internal,},FirstTimestamp:2026-07-30 03:45:19.075781424 +0000 UTC m=+0.378578137,LastTimestamp:2026-07-30 03:45:19.075781424 +0000 UTC m=+0.378578137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-114.ec2.internal,}" Jul 30 03:45:19.092545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.092368 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vz2c6" Jul 30 03:45:19.097063 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.097026 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 03:45:19.097227 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.097181 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 03:45:19.097917 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.097892 2568 manager.go:354] Recovery completed Jul 30 03:45:19.103665 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.103651 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.105758 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.105738 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.105832 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.105774 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.105832 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.105787 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.106398 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.106371 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 03:45:19.106398 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.106394 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 03:45:19.106508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.106402 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.106508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.106414 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 30 03:45:19.107212 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.107184 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.107263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.107213 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.107263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.107226 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.108241 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.108226 2568 policy_none.go:49] "None policy: Start" Jul 30 03:45:19.108277 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.108247 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 03:45:19.108277 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.108259 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 03:45:19.109642 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.109631 2568 policy_none.go:47] "Start" Jul 30 03:45:19.151781 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.151763 2568 manager.go:337] "Starting Device Plugin manager" Jul 30 03:45:19.151879 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.151804 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 03:45:19.151879 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.151816 2568 server.go:87] "Starting device plugin registration server" Jul 30 03:45:19.152091 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.152078 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 03:45:19.152136 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.152093 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 03:45:19.152197 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.152185 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 03:45:19.152273 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.152262 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 03:45:19.152321 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.152273 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 03:45:19.152919 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.152893 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 03:45:19.153003 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.152951 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.252448 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.252405 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.253917 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.253417 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.253917 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.253454 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.253917 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.253465 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.253917 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.253492 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.262118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.262071 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.262118 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.262096 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-114.ec2.internal\": node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.272896 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.272878 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.302335 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.302304 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 03:45:19.303688 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.303656 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 03:45:19.303688 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.303694 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 03:45:19.303844 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.303715 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 03:45:19.303844 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.303747 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 03:45:19.305808 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.305789 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 03:45:19.373836 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.373785 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.404275 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.404241 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal"] Jul 30 03:45:19.404412 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.404334 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.406144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.406123 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.406262 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.406162 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.406262 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.406199 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.407637 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.407624 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.407742 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.407727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.407814 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.407761 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.408434 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408411 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.408516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408442 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.408516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408461 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.408516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408472 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.408516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408496 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.408516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.408510 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.409805 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.409790 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.409870 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.409817 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 03:45:19.411335 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.411319 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientMemory" Jul 30 03:45:19.411479 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.411469 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 03:45:19.411557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.411550 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeHasSufficientPID" Jul 30 03:45:19.439155 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.439124 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-114.ec2.internal\" not found" node="ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.443455 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.443439 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-114.ec2.internal\" not found" node="ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.474879 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.474847 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.486154 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.486122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.486291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.486155 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.486291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.486222 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12140044dd1a0f62cf96e929cee8dd45-config\") pod \"kube-apiserver-proxy-ip-10-0-134-114.ec2.internal\" (UID: \"12140044dd1a0f62cf96e929cee8dd45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.575336 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.575269 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.586617 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12140044dd1a0f62cf96e929cee8dd45-config\") pod \"kube-apiserver-proxy-ip-10-0-134-114.ec2.internal\" (UID: \"12140044dd1a0f62cf96e929cee8dd45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.586682 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.586682 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.586747 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12140044dd1a0f62cf96e929cee8dd45-config\") pod \"kube-apiserver-proxy-ip-10-0-134-114.ec2.internal\" (UID: \"12140044dd1a0f62cf96e929cee8dd45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.586747 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.586747 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.586683 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c960ce124b33299386a930b2630663c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal\" (UID: \"c960ce124b33299386a930b2630663c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.675765 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.675719 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:19.742285 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.742256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.746201 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:19.746162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:19.776717 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:19.776675 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.134683 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.134589 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.235107 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.235062 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.281649 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.281617 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 03:45:21.282159 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.281796 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 03:45:21.336229 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.336188 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.358996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.358953 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 03:40:20 +0000 UTC" deadline="2028-04-20 05:19:33.277705092 +0000 UTC" Jul 30 03:45:21.358996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.358982 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15121h34m11.91872594s" Jul 30 03:45:21.365911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.365876 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 03:45:21.436865 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.436781 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.438945 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.438929 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 03:45:21.537948 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.537906 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.639037 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:21.639010 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-114.ec2.internal\" not found" Jul 30 03:45:21.644005 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:21.643972 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc960ce124b33299386a930b2630663c4.slice/crio-24e29a3687ee8623d55387b4cf6e7aabbabc7767c8bdc8954460ea690d83425d WatchSource:0}: Error finding container 24e29a3687ee8623d55387b4cf6e7aabbabc7767c8bdc8954460ea690d83425d: Status 404 returned error can't find the container with id 24e29a3687ee8623d55387b4cf6e7aabbabc7767c8bdc8954460ea690d83425d Jul 30 03:45:21.648783 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.648763 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 03:45:21.653359 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:21.653334 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12140044dd1a0f62cf96e929cee8dd45.slice/crio-89e23afc47f4bfd9e84f0b8f5fc5ad604c0c746f6fbecc42c577017e65b5d8e0 WatchSource:0}: Error finding container 89e23afc47f4bfd9e84f0b8f5fc5ad604c0c746f6fbecc42c577017e65b5d8e0: Status 404 returned error can't find the container with id 89e23afc47f4bfd9e84f0b8f5fc5ad604c0c746f6fbecc42c577017e65b5d8e0 Jul 30 03:45:21.660043 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.660025 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 03:45:21.742791 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.742716 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" Jul 30 03:45:21.751908 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.751788 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 03:45:21.753151 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.752975 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" Jul 30 03:45:21.761573 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:21.761554 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 03:45:22.204235 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.204149 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 03:45:22.322286 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.322022 2568 apiserver.go:52] "Watching apiserver" Jul 30 03:45:22.330042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.330014 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 03:45:22.330564 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.330535 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-xrnvf","openshift-image-registry/node-ca-wn5lb","openshift-multus/multus-g2xcl","openshift-multus/network-metrics-daemon-8tb9g","openshift-network-diagnostics/network-check-target-vdg9f","openshift-network-operator/iptables-alerter-2pd6d","kube-system/global-pull-secret-syncer-ch58f","kube-system/konnectivity-agent-znthr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal","openshift-multus/multus-additional-cni-plugins-6vrb7","openshift-ovn-kubernetes/ovnkube-node-kpn5x","kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9bw4b"] Jul 30 03:45:22.332579 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.332559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.332681 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.332637 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:22.333919 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.333902 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.335248 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.335226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.338684 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.338661 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 03:45:22.339152 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339130 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 03:45:22.339566 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339318 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.339566 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339346 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dn44q\"" Jul 30 03:45:22.339566 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.339383 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:22.339566 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339456 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:22.339566 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.339515 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:22.339840 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339640 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 03:45:22.339840 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339816 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.339840 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339828 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fr24d\"" Jul 30 03:45:22.340062 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.339983 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.340062 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.340024 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 03:45:22.340492 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.340455 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.340965 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.340924 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.342429 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.342408 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 03:45:22.342537 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.342522 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 03:45:22.342537 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.342532 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.342684 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.342635 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.343196 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.343179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.345241 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.345741 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4p6c\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.345930 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.345989 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.346155 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87lxp\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.346021 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-drcks\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.346293 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.346395 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.346534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.346407 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.347667 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.347646 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rrb25\"" Jul 30 03:45:22.349962 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.347651 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 03:45:22.353359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.353338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.353783 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.353761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.354041 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354014 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.354121 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-cnibin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354121 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-os-release\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354223 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-host-slash\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.354273 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-socket-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.354323 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-host\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.354376 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdbz\" (UniqueName: \"kubernetes.io/projected/e768cf7a-34a7-4fcb-875f-759fd9a6d489-kube-api-access-ccdbz\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.354376 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-kubelet-config\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.354485 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.354485 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-cnibin\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.354485 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-netns\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354630 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354508 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysconfig\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.354630 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354553 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-socket-dir-parent\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354630 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9hk\" (UniqueName: \"kubernetes.io/projected/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-kube-api-access-9b9hk\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.354630 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354605 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlx5w\" (UniqueName: \"kubernetes.io/projected/2048e973-464a-4cca-a4e1-7976f68c5c04-kube-api-access-xlx5w\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9sh\" (UniqueName: \"kubernetes.io/projected/f9ed8dbc-9e70-479f-b56f-988991b5c082-kube-api-access-lk9sh\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-bin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-iptables-alerter-script\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-sys\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-k8s-cni-cncf-io\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.354806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-conf\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-systemd\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-var-lib-kubelet\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-daemon-config\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-os-release\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-registration-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-kubelet\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-lib-modules\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.354972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-sys-fs\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-hostroot\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-conf-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-system-cni-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.355117 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-tuned\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-tmp\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-cni-binary-copy\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355200 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-multus-certs\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-kubernetes\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355238 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vj22\" (UniqueName: \"kubernetes.io/projected/2df5d138-642c-4381-b83b-6ab0e3520c87-kube-api-access-2vj22\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-system-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-multus\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-etc-kubernetes\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-device-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355366 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxbg\" (UniqueName: \"kubernetes.io/projected/de93d457-c24b-40d8-8c37-2918bb3c691e-kube-api-access-bvxbg\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-run\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-dbus\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.355856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.356646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-modprobe-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.356646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.355754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.356837 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.356818 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.358207 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358188 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.358356 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358344 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7gjf\"" Jul 30 03:45:22.358499 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358481 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.358596 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358577 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 03:45:22.358767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358755 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 03:45:22.358937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358915 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 03:45:22.359021 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.358995 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 03:45:22.359074 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.359021 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 03:45:22.359443 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.359425 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 03:45:22.359814 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.359796 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 03:45:22.360123 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.360103 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hvhkn\"" Jul 30 03:45:22.360312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.360293 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 03:45:22.360611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.360593 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wcz7j\"" Jul 30 03:45:22.362760 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.362267 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 03:45:22.373713 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.373692 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpc25" Jul 30 03:45:22.380925 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.380514 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpc25" Jul 30 03:45:22.444070 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.443990 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmvp\" (UniqueName: \"kubernetes.io/projected/8a801077-44f7-4e42-a883-50d978b8909e-kube-api-access-7gmvp\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-k8s-cni-cncf-io\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456754 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-conf\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-netd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-systemd\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-var-lib-kubelet\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-daemon-config\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-os-release\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-registration-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2k5\" (UniqueName: \"kubernetes.io/projected/f1c50bcc-64b6-4f18-8187-748c1e69fb49-kube-api-access-lb2k5\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.456992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-kubelet\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-lib-modules\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb61c933-717d-436b-a002-23b4f6f8a88e-agent-certs\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.459251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457064 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-env-overrides\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29eb6fec-b740-4082-a3c7-00be532eb15a-ovn-node-metrics-cert\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-script-lib\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457135 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gm6\" (UniqueName: \"kubernetes.io/projected/29eb6fec-b740-4082-a3c7-00be532eb15a-kube-api-access-67gm6\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-sys-fs\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a801077-44f7-4e42-a883-50d978b8909e-tmp-dir\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-systemd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-node-log\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-hostroot\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-conf-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-system-cni-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-tuned\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-tmp\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-etc-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.460488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-cni-binary-copy\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-multus-certs\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-kubelet\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-slash\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-kubernetes\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vj22\" (UniqueName: \"kubernetes.io/projected/2df5d138-642c-4381-b83b-6ab0e3520c87-kube-api-access-2vj22\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-system-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-multus\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-etc-kubernetes\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-device-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-bin\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxbg\" (UniqueName: \"kubernetes.io/projected/de93d457-c24b-40d8-8c37-2918bb3c691e-kube-api-access-bvxbg\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-run\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.461567 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb61c933-717d-436b-a002-23b4f6f8a88e-konnectivity-ca\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.457985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-netns\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-log-socket\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-dbus\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-modprobe-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-cnibin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-os-release\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-host-slash\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-socket-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-host\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdbz\" (UniqueName: \"kubernetes.io/projected/e768cf7a-34a7-4fcb-875f-759fd9a6d489-kube-api-access-ccdbz\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-systemd-units\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-kubelet-config\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.462276 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-cnibin\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-netns\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysconfig\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-ovn\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-socket-dir-parent\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9hk\" (UniqueName: \"kubernetes.io/projected/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-kube-api-access-9b9hk\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlx5w\" (UniqueName: \"kubernetes.io/projected/2048e973-464a-4cca-a4e1-7976f68c5c04-kube-api-access-xlx5w\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458619 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a801077-44f7-4e42-a883-50d978b8909e-hosts-file\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-var-lib-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-config\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9sh\" (UniqueName: \"kubernetes.io/projected/f9ed8dbc-9e70-479f-b56f-988991b5c082-kube-api-access-lk9sh\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-bin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458754 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-iptables-alerter-script\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-sys\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c50bcc-64b6-4f18-8187-748c1e69fb49-host\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1c50bcc-64b6-4f18-8187-748c1e69fb49-serviceca\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458981 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.463250 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.458986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-system-cni-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.459041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-k8s-cni-cncf-io\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.459065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-kubelet-config\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.459079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-socket-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.459109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-conf-dir\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.459045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-host\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.459418 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.459514 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:22.959491878 +0000 UTC m=+3.004991068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.460719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.460807 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-registration-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-bin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-run\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461320 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-sys-fs\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-iptables-alerter-script\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-sys\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461695 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461738 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-cnibin\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b45e529-005c-4897-890c-6c53db7030d3-dbus\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.464108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.461989 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.462028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-host-slash\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.462065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-os-release\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.462754 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysconfig\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.462997 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-cni-multus\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-systemd\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-cnibin\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-etc-kubernetes\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463154 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-netns\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-conf\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463753 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-daemon-config\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.463822 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-var-lib-kubelet\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.464204 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.464256 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-run-multus-certs\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.464667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de93d457-c24b-40d8-8c37-2918bb3c691e-cni-binary-copy\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.464747 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-os-release\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.464996 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.464850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-sysctl-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465236 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9ed8dbc-9e70-479f-b56f-988991b5c082-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465326 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-hostroot\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.465500 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.465550 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:22.965531183 +0000 UTC m=+3.011030358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-modprobe-d\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.465753 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465663 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2048e973-464a-4cca-a4e1-7976f68c5c04-device-dir\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.466142 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-multus-socket-dir-parent\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.466142 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.465994 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-lib-modules\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.466142 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.466032 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de93d457-c24b-40d8-8c37-2918bb3c691e-host-var-lib-kubelet\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.466142 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.466066 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9ed8dbc-9e70-479f-b56f-988991b5c082-system-cni-dir\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.466142 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.466118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-kubernetes\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.467669 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.467647 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:22.467801 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.467681 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:22.467801 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.467695 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:22.467801 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.467758 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:22.967741098 +0000 UTC m=+3.013240274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:22.468258 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.468237 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-etc-tuned\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.477705 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e768cf7a-34a7-4fcb-875f-759fd9a6d489-tmp\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.478077 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vj22\" (UniqueName: \"kubernetes.io/projected/2df5d138-642c-4381-b83b-6ab0e3520c87-kube-api-access-2vj22\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.478099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdbz\" (UniqueName: \"kubernetes.io/projected/e768cf7a-34a7-4fcb-875f-759fd9a6d489-kube-api-access-ccdbz\") pod \"tuned-9bw4b\" (UID: \"e768cf7a-34a7-4fcb-875f-759fd9a6d489\") " pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.478107 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9sh\" (UniqueName: \"kubernetes.io/projected/f9ed8dbc-9e70-479f-b56f-988991b5c082-kube-api-access-lk9sh\") pod \"multus-additional-cni-plugins-6vrb7\" (UID: \"f9ed8dbc-9e70-479f-b56f-988991b5c082\") " pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.478113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxbg\" (UniqueName: \"kubernetes.io/projected/de93d457-c24b-40d8-8c37-2918bb3c691e-kube-api-access-bvxbg\") pod \"multus-g2xcl\" (UID: \"de93d457-c24b-40d8-8c37-2918bb3c691e\") " pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.479157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.478137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9hk\" (UniqueName: \"kubernetes.io/projected/e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0-kube-api-access-9b9hk\") pod \"iptables-alerter-2pd6d\" (UID: \"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0\") " pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.481871 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.481847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlx5w\" (UniqueName: \"kubernetes.io/projected/2048e973-464a-4cca-a4e1-7976f68c5c04-kube-api-access-xlx5w\") pod \"aws-ebs-csi-driver-node-tpgcw\" (UID: \"2048e973-464a-4cca-a4e1-7976f68c5c04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.559715 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-kubelet\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559715 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-slash\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-bin\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb61c933-717d-436b-a002-23b4f6f8a88e-konnectivity-ca\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-netns\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-log-socket\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-systemd-units\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-ovn\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.559911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a801077-44f7-4e42-a883-50d978b8909e-hosts-file\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-var-lib-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-config\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c50bcc-64b6-4f18-8187-748c1e69fb49-host\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1c50bcc-64b6-4f18-8187-748c1e69fb49-serviceca\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.559997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmvp\" (UniqueName: \"kubernetes.io/projected/8a801077-44f7-4e42-a883-50d978b8909e-kube-api-access-7gmvp\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-netd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2k5\" (UniqueName: \"kubernetes.io/projected/f1c50bcc-64b6-4f18-8187-748c1e69fb49-kube-api-access-lb2k5\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb61c933-717d-436b-a002-23b4f6f8a88e-agent-certs\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-env-overrides\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29eb6fec-b740-4082-a3c7-00be532eb15a-ovn-node-metrics-cert\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-script-lib\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gm6\" (UniqueName: \"kubernetes.io/projected/29eb6fec-b740-4082-a3c7-00be532eb15a-kube-api-access-67gm6\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a801077-44f7-4e42-a883-50d978b8909e-tmp-dir\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-systemd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-node-log\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.560283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-etc-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560274 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-etc-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-kubelet\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560327 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-slash\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560355 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-bin\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-systemd-units\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560605 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-cni-netd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-ovn\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-log-socket\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560837 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-run-netns\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560855 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb61c933-717d-436b-a002-23b4f6f8a88e-konnectivity-ca\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.560999 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a801077-44f7-4e42-a883-50d978b8909e-hosts-file\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561038 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a801077-44f7-4e42-a883-50d978b8909e-tmp-dir\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-var-lib-openvswitch\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561052 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-script-lib\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-run-systemd\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561249 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c50bcc-64b6-4f18-8187-748c1e69fb49-host\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1c50bcc-64b6-4f18-8187-748c1e69fb49-serviceca\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-ovnkube-config\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561668 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-node-log\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.561774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29eb6fec-b740-4082-a3c7-00be532eb15a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.562068 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.561869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29eb6fec-b740-4082-a3c7-00be532eb15a-env-overrides\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.564538 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.564495 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29eb6fec-b740-4082-a3c7-00be532eb15a-ovn-node-metrics-cert\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.564538 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.564493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb61c933-717d-436b-a002-23b4f6f8a88e-agent-certs\") pod \"konnectivity-agent-znthr\" (UID: \"cb61c933-717d-436b-a002-23b4f6f8a88e\") " pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.567333 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.567283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" event={"ID":"12140044dd1a0f62cf96e929cee8dd45","Type":"ContainerStarted","Data":"89e23afc47f4bfd9e84f0b8f5fc5ad604c0c746f6fbecc42c577017e65b5d8e0"} Jul 30 03:45:22.569598 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.569524 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmvp\" (UniqueName: \"kubernetes.io/projected/8a801077-44f7-4e42-a883-50d978b8909e-kube-api-access-7gmvp\") pod \"node-resolver-xrnvf\" (UID: \"8a801077-44f7-4e42-a883-50d978b8909e\") " pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.569826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.569807 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gm6\" (UniqueName: \"kubernetes.io/projected/29eb6fec-b740-4082-a3c7-00be532eb15a-kube-api-access-67gm6\") pod \"ovnkube-node-kpn5x\" (UID: \"29eb6fec-b740-4082-a3c7-00be532eb15a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.571899 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.571875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2k5\" (UniqueName: \"kubernetes.io/projected/f1c50bcc-64b6-4f18-8187-748c1e69fb49-kube-api-access-lb2k5\") pod \"node-ca-wn5lb\" (UID: \"f1c50bcc-64b6-4f18-8187-748c1e69fb49\") " pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.571993 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.571904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" event={"ID":"c960ce124b33299386a930b2630663c4","Type":"ContainerStarted","Data":"24e29a3687ee8623d55387b4cf6e7aabbabc7767c8bdc8954460ea690d83425d"} Jul 30 03:45:22.655560 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.655518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" Jul 30 03:45:22.666671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.666646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2xcl" Jul 30 03:45:22.676186 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.676161 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pd6d" Jul 30 03:45:22.683830 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.683811 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" Jul 30 03:45:22.692578 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.692551 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" Jul 30 03:45:22.702394 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.702371 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wn5lb" Jul 30 03:45:22.710286 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.710225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:22.716947 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.716929 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:22.722667 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.722650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrnvf" Jul 30 03:45:22.962830 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:22.962743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:22.962963 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.962905 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:22.962999 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:22.962974 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:23.962957277 +0000 UTC m=+4.008456449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:23.063908 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:23.063875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:23.063928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064039 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064064 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064091 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064104 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:24.064087511 +0000 UTC m=+4.109586683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:23.064109 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064107 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:23.064418 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.064163 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:24.064146344 +0000 UTC m=+4.109645518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:23.383479 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:23.383434 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:40:22 +0000 UTC" deadline="2028-03-31 22:43:51.105904091 +0000 UTC" Jul 30 03:45:23.383479 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:23.383478 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14658h58m27.722447083s" Jul 30 03:45:23.626012 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:23.625979 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde93d457_c24b_40d8_8c37_2918bb3c691e.slice/crio-361ed7af0e93f5ec0ceb2d5e48ebcadfb7606d36e4939e127e58da84823dd327 WatchSource:0}: Error finding container 361ed7af0e93f5ec0ceb2d5e48ebcadfb7606d36e4939e127e58da84823dd327: Status 404 returned error can't find the container with id 361ed7af0e93f5ec0ceb2d5e48ebcadfb7606d36e4939e127e58da84823dd327 Jul 30 03:45:23.627501 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:23.627451 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29eb6fec_b740_4082_a3c7_00be532eb15a.slice/crio-e00225e45c30fa14e81c42b7cd9c5d39d8e6865d64fe3321a54852c35e924252 WatchSource:0}: Error finding container e00225e45c30fa14e81c42b7cd9c5d39d8e6865d64fe3321a54852c35e924252: Status 404 returned error can't find the container with id e00225e45c30fa14e81c42b7cd9c5d39d8e6865d64fe3321a54852c35e924252 Jul 30 03:45:23.629190 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:45:23.629165 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c50bcc_64b6_4f18_8187_748c1e69fb49.slice/crio-89d4349129e42c740664a57e5b261ed077974a6b87138df456b47fcc0581ef64 WatchSource:0}: Error finding container 89d4349129e42c740664a57e5b261ed077974a6b87138df456b47fcc0581ef64: Status 404 returned error can't find the container with id 89d4349129e42c740664a57e5b261ed077974a6b87138df456b47fcc0581ef64 Jul 30 03:45:23.970254 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:23.970000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:23.970415 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.970178 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:23.970415 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:23.970345 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:25.970328797 +0000 UTC m=+6.015827973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:24.070763 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.070726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:24.070934 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.070791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:24.070934 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.070910 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:24.071056 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.070972 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:26.07095338 +0000 UTC m=+6.116452555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:24.071386 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.071364 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:24.071543 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.071393 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:24.071543 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.071406 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:24.071543 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.071451 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:26.071436492 +0000 UTC m=+6.116935666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:24.384085 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.383985 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:40:22 +0000 UTC" deadline="2028-03-30 18:48:50.62436423 +0000 UTC" Jul 30 03:45:24.384085 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.384048 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14631h3m26.240320944s" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.564518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.564645 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.565023 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.565112 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.565175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:24.565303 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:24.565257 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:24.606374 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.606336 2568 generic.go:356] "Generic (PLEG): container finished" podID="c960ce124b33299386a930b2630663c4" containerID="3fd783fed7cfcb356c98174544e64747195a6101feaef0473c2d9f7db8eb4e12" exitCode=0 Jul 30 03:45:24.606581 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.606436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" event={"ID":"c960ce124b33299386a930b2630663c4","Type":"ContainerDied","Data":"3fd783fed7cfcb356c98174544e64747195a6101feaef0473c2d9f7db8eb4e12"} Jul 30 03:45:24.622036 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.622005 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" event={"ID":"2048e973-464a-4cca-a4e1-7976f68c5c04","Type":"ContainerStarted","Data":"3bfa9d80cdf10fe1f266e2d0ce7e4b39fb788b6222b4d9d48b6478e0e7c6e00e"} Jul 30 03:45:24.643870 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.643790 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wn5lb" event={"ID":"f1c50bcc-64b6-4f18-8187-748c1e69fb49","Type":"ContainerStarted","Data":"89d4349129e42c740664a57e5b261ed077974a6b87138df456b47fcc0581ef64"} Jul 30 03:45:24.657354 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.657303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2xcl" event={"ID":"de93d457-c24b-40d8-8c37-2918bb3c691e","Type":"ContainerStarted","Data":"361ed7af0e93f5ec0ceb2d5e48ebcadfb7606d36e4939e127e58da84823dd327"} Jul 30 03:45:24.666938 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.666900 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" event={"ID":"12140044dd1a0f62cf96e929cee8dd45","Type":"ContainerStarted","Data":"d23ba7dc49baf09d683b6eed72684d9db8dc6bc432b6f30c79b07f928ea2e84e"} Jul 30 03:45:24.683855 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.683792 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-114.ec2.internal" podStartSLOduration=3.683772385 podStartE2EDuration="3.683772385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:24.683260986 +0000 UTC m=+4.728760182" watchObservedRunningTime="2026-07-30 03:45:24.683772385 +0000 UTC m=+4.729271580" Jul 30 03:45:24.691448 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.691412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znthr" event={"ID":"cb61c933-717d-436b-a002-23b4f6f8a88e","Type":"ContainerStarted","Data":"3d69bc8230cf0265d368e8166b15d33ac04109962055bb1b21007b373c65525f"} Jul 30 03:45:24.696260 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.696190 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrnvf" event={"ID":"8a801077-44f7-4e42-a883-50d978b8909e","Type":"ContainerStarted","Data":"632f48370444afa3ed68fdacaf8df631cb80422ba7c0c319136feba927dbb5a8"} Jul 30 03:45:24.709363 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.709328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"e00225e45c30fa14e81c42b7cd9c5d39d8e6865d64fe3321a54852c35e924252"} Jul 30 03:45:24.728981 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.728942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pd6d" event={"ID":"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0","Type":"ContainerStarted","Data":"dbda39e0349d65838d37434fd31fbc682a99534480cec08d4b377a6839212c75"} Jul 30 03:45:24.749367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.749333 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerStarted","Data":"865360d73cdee5651bedabf909473448a8b85d4d2073fd3372110858d42f439e"} Jul 30 03:45:24.765251 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:24.763751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" event={"ID":"e768cf7a-34a7-4fcb-875f-759fd9a6d489","Type":"ContainerStarted","Data":"cdd15889684e3d46e669ed3a5cb33ef6186144b918487c469c094faf8f0aa89a"} Jul 30 03:45:25.784804 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:25.784752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" event={"ID":"c960ce124b33299386a930b2630663c4","Type":"ContainerStarted","Data":"2980700e977bf8d417b73679b6de7f298f256fbfb70512088e72f1ccab65f9ac"} Jul 30 03:45:25.989159 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:25.989120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:25.989328 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:25.989302 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:25.989401 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:25.989368 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:29.989348157 +0000 UTC m=+10.034847343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:26.090282 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:26.090200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:26.090455 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:26.090285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:26.090455 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090425 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:26.090455 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090442 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:26.090455 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090456 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:26.090729 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090536 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:30.090516447 +0000 UTC m=+10.136015634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:26.090957 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090923 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:26.091036 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.090990 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:30.090974451 +0000 UTC m=+10.136473627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:26.561658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:26.561578 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:26.561867 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.561705 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:26.562144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:26.562101 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:26.562437 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.562261 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:26.562437 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:26.562320 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:26.562437 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:26.562384 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:28.562668 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:28.562579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:28.562668 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:28.562609 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:28.563047 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:28.562579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:28.563047 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:28.562721 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:28.563047 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:28.562803 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:28.563047 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:28.562890 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:30.024128 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.024081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:30.024616 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.024228 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:30.024616 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.024295 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:38.024274068 +0000 UTC m=+18.069773241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:30.125059 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.125018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:30.125259 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.125098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:30.125259 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125207 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:30.125259 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125226 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:30.125259 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125243 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:30.125259 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125256 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:30.125578 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125286 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:38.125264754 +0000 UTC m=+18.170764014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:30.125578 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.125313 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:38.125295399 +0000 UTC m=+18.170794587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:30.562841 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.562806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:30.563025 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.562907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:30.563025 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:30.562940 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:30.563025 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.562959 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:30.563025 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.562997 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:30.563229 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:30.563059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:32.561631 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:32.561591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:32.562093 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:32.561594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:32.562093 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:32.561742 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:32.562093 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:32.561784 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:32.562093 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:32.561593 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:32.562093 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:32.561859 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:34.561643 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:34.561607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:34.562052 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:34.561607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:34.562052 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:34.561716 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:34.562052 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:34.561797 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:34.562052 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:34.561607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:34.562052 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:34.561870 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:36.564013 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:36.563982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:36.564013 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:36.564018 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:36.564459 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:36.564101 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:36.564459 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:36.564139 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:36.564459 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:36.564193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:36.564459 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:36.564308 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:38.089199 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.089162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:38.089780 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.089290 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:38.089780 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.089350 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:54.089331342 +0000 UTC m=+34.134830517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:38.190249 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.190213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:38.190432 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.190291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:38.190432 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190385 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:38.190547 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190431 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:38.190547 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190454 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:38.190547 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190458 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:54.190437287 +0000 UTC m=+34.235936462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:38.190547 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190485 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:38.190547 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.190530 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:54.190518814 +0000 UTC m=+34.236017990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:38.565207 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.565131 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:38.565354 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.565132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:38.565354 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.565242 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:38.565354 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.565309 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:38.565354 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:38.565130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:38.565531 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:38.565446 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:40.564545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:40.564509 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:40.564987 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:40.564619 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:40.564987 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:40.564509 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:40.564987 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:40.564695 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:40.564987 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:40.564509 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:40.564987 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:40.564750 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:42.593095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.592872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:42.593482 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:42.593204 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:42.593961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.593653 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:42.593961 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:42.593763 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:42.593961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.593818 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:42.593961 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:42.593890 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:42.818880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.818586 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znthr" event={"ID":"cb61c933-717d-436b-a002-23b4f6f8a88e","Type":"ContainerStarted","Data":"9866a7bd7c1ca789efb611e399f7a31827f0b57f961d922346df4c6c74165722"} Jul 30 03:45:42.820037 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.820012 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrnvf" event={"ID":"8a801077-44f7-4e42-a883-50d978b8909e","Type":"ContainerStarted","Data":"bb6b1779805f21eb651ef20efba0fbb10c5546b4e58fadc05d6f495fe3193553"} Jul 30 03:45:42.821888 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.821863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"819c6fb4fabd0ed9959789fbaade6b368ef4ea71ae35bc394acf74c7ab487bb2"} Jul 30 03:45:42.822084 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.822058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"53fab88ead53b610ea5ac892d4e17c71631f3a74a2745633a3ddadbc6ef8f6ea"} Jul 30 03:45:42.823649 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.823622 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="d8a6bc1b6d3f981978c2b6b910d9640110247887e8af150e138f1fbdb30b2e56" exitCode=0 Jul 30 03:45:42.823759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.823704 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"d8a6bc1b6d3f981978c2b6b910d9640110247887e8af150e138f1fbdb30b2e56"} Jul 30 03:45:42.825589 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.825570 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" event={"ID":"e768cf7a-34a7-4fcb-875f-759fd9a6d489","Type":"ContainerStarted","Data":"77610567a78a73206d47a58bf876a810d93f367d0fb57015eee424ea28caa23e"} Jul 30 03:45:42.826880 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.826865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" event={"ID":"2048e973-464a-4cca-a4e1-7976f68c5c04","Type":"ContainerStarted","Data":"158ac92ef9b60c895f103907b705d194f302c61a2bcec8e5c1a819b5d7b022fb"} Jul 30 03:45:42.829053 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.829032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wn5lb" event={"ID":"f1c50bcc-64b6-4f18-8187-748c1e69fb49","Type":"ContainerStarted","Data":"185b72e77ef0a0b5b72dd37a3fedd81a84a4c279a14b4be7c9ab45bdcbe1f07f"} Jul 30 03:45:42.832742 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.832702 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-znthr" podStartSLOduration=4.017544733 podStartE2EDuration="22.832687456s" totalImagesPullingTime="18.815142723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:42.832565387 +0000 UTC m=+22.878064583" watchObservedRunningTime="2026-07-30 03:45:42.832687456 +0000 UTC m=+22.878186652" Jul 30 03:45:42.832844 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.832830 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-114.ec2.internal" podStartSLOduration=21.832824437 podStartE2EDuration="21.832824437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:25.799144859 +0000 UTC m=+5.844644055" watchObservedRunningTime="2026-07-30 03:45:42.832824437 +0000 UTC m=+22.878323632" Jul 30 03:45:42.839934 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.839908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2xcl" event={"ID":"de93d457-c24b-40d8-8c37-2918bb3c691e","Type":"ContainerStarted","Data":"916ed9b533b697e24bab62dcadb597a47e039852313d062bf8d6afafcd18da65"} Jul 30 03:45:42.848132 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.848080 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9bw4b" podStartSLOduration=3.974158704 podStartE2EDuration="22.848060175s" totalImagesPullingTime="18.873901471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:42.847145497 +0000 UTC m=+22.892644693" watchObservedRunningTime="2026-07-30 03:45:42.848060175 +0000 UTC m=+22.893559371" Jul 30 03:45:42.860818 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.860771 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wn5lb" podStartSLOduration=4.012654327 podStartE2EDuration="22.86075676s" totalImagesPullingTime="18.848102433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:42.860409939 +0000 UTC m=+22.905909133" watchObservedRunningTime="2026-07-30 03:45:42.86075676 +0000 UTC m=+22.906255955" Jul 30 03:45:42.874786 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.874737 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xrnvf" podStartSLOduration=4.032950678 podStartE2EDuration="22.874716113s" totalImagesPullingTime="18.841765435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:42.87414535 +0000 UTC m=+22.919644567" watchObservedRunningTime="2026-07-30 03:45:42.874716113 +0000 UTC m=+22.920215312" Jul 30 03:45:42.913668 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:42.913623 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g2xcl" podStartSLOduration=3.994883711 podStartE2EDuration="22.913611414s" totalImagesPullingTime="18.918727703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:42.913491254 +0000 UTC m=+22.958990450" watchObservedRunningTime="2026-07-30 03:45:42.913611414 +0000 UTC m=+22.959110608" Jul 30 03:45:43.634400 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.634361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:43.635220 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.635037 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:43.845534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.845412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"3a704f90fdbb3d926e5796a27bf7d84581cffd5579af28b8dc0a3ef5f8ae45ca"} Jul 30 03:45:43.845739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.845549 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"99c2c350fd42c12f66d19fb040d1cffeb90e06ed7eeec8057f51f2f9c26f9805"} Jul 30 03:45:43.845739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.845590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"1b798e67fa606c974f91769f6641b9fb4bd91919376f37be57f63101850743fe"} Jul 30 03:45:43.845739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.845636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"dd33f1c5576f5d3a5563db65cf278e726839227a48e121457576b932ac916c22"} Jul 30 03:45:43.848310 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.848291 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:43.849361 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:43.849323 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-znthr" Jul 30 03:45:44.070360 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.070331 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 03:45:44.425887 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.425783 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T03:45:44.070355858Z","UUID":"17bf550d-5d0a-401b-af7d-65a7e3e0ed3d","Handler":null,"Name":"","Endpoint":""} Jul 30 03:45:44.427330 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.427298 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 03:45:44.427510 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.427343 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 03:45:44.565358 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.565329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:44.565579 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:44.565550 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:44.565713 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.565652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:44.565778 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:44.565755 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:44.565819 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.565793 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:44.565873 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:44.565859 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:44.850343 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.850248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" event={"ID":"2048e973-464a-4cca-a4e1-7976f68c5c04","Type":"ContainerStarted","Data":"25c8c2ccc88bcda6363d7a2719eefc1aea745f041a3050d0fe06c706390bf8af"} Jul 30 03:45:44.852860 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:44.852829 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pd6d" event={"ID":"e8fd686d-5f9d-47ec-a28a-3c4e9f03b5e0","Type":"ContainerStarted","Data":"43de051ce8649bb08a0acc557e09f6b568ec7ba84be032b0691af4c803126c43"} Jul 30 03:45:45.858566 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:45.858441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"21c31410e478d9ee227536accbf31ad6e23e1a02ed39eec2526eb907e58318a2"} Jul 30 03:45:45.860750 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:45.860721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" event={"ID":"2048e973-464a-4cca-a4e1-7976f68c5c04","Type":"ContainerStarted","Data":"168dcbc6480372bdb0dacf2478f60359e985609cf69866c8f51826e11f6d35f8"} Jul 30 03:45:45.882953 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:45.882880 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2pd6d" podStartSLOduration=7.020862197 podStartE2EDuration="25.882862588s" totalImagesPullingTime="18.862000391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:44.867313848 +0000 UTC m=+24.912813059" watchObservedRunningTime="2026-07-30 03:45:45.882862588 +0000 UTC m=+25.928361792" Jul 30 03:45:45.883116 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:45.883078 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpgcw" podStartSLOduration=4.751977658 podStartE2EDuration="25.883065387s" totalImagesPullingTime="21.131087729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:45.882320917 +0000 UTC m=+25.927820111" watchObservedRunningTime="2026-07-30 03:45:45.883065387 +0000 UTC m=+25.928564587" Jul 30 03:45:46.561945 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:46.561901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:46.562131 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:46.561921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:46.562131 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:46.562053 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:46.562221 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:46.562146 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:46.562221 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:46.561932 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:46.562310 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:46.562232 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:47.868486 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.868174 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" event={"ID":"29eb6fec-b740-4082-a3c7-00be532eb15a","Type":"ContainerStarted","Data":"e8f15393b789e8f5aa5366e0bddac9d585d93b31b783fa5724bb4761728eff0e"} Jul 30 03:45:47.869271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.868504 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:47.870212 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.870182 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="487defc924a88ab099f02bf0a3e0d07eabb8f6cd6bd64049c87b2cc40d8a73be" exitCode=0 Jul 30 03:45:47.870347 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.870217 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"487defc924a88ab099f02bf0a3e0d07eabb8f6cd6bd64049c87b2cc40d8a73be"} Jul 30 03:45:47.885397 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.885368 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:47.892270 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.892223 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" podStartSLOduration=8.980016442 podStartE2EDuration="27.892207064s" totalImagesPullingTime="18.912190622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:47.891972156 +0000 UTC m=+27.937471350" watchObservedRunningTime="2026-07-30 03:45:47.892207064 +0000 UTC m=+27.937706258" Jul 30 03:45:47.899301 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:47.899284 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xrnvf_8a801077-44f7-4e42-a883-50d978b8909e/dns-node-resolver/0.log" Jul 30 03:45:48.564832 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.564759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:48.565025 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.564884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:48.565025 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.564897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:48.565025 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:48.564962 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:48.565194 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:48.565043 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:48.565194 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:48.565074 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:48.872358 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.872277 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:48.872795 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.872363 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:48.887317 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:48.887294 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:45:49.281892 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.281867 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wn5lb_f1c50bcc-64b6-4f18-8187-748c1e69fb49/node-ca/0.log" Jul 30 03:45:49.511493 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.511401 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch58f"] Jul 30 03:45:49.511707 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.511545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:49.511707 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:49.511666 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:49.514528 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.514503 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdg9f"] Jul 30 03:45:49.514647 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.514589 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:49.514696 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:49.514666 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:49.515036 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.515016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8tb9g"] Jul 30 03:45:49.515116 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.515101 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:49.515258 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:49.515240 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:49.876231 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.876132 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="e92b763bb68496987eb133c670eed3e68c0aa7ce4ee56380f0328c17fb8d0f8e" exitCode=0 Jul 30 03:45:49.876231 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:49.876167 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"e92b763bb68496987eb133c670eed3e68c0aa7ce4ee56380f0328c17fb8d0f8e"} Jul 30 03:45:51.562200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:51.561996 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:51.562698 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:51.562073 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:51.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:51.562310 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:51.562698 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:51.562081 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:51.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:51.562342 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:51.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:51.562436 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:51.883893 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:51.883820 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="6e8dff467e082a6a0a903480bec9efc0229742de270c9c4393656c8db17084fa" exitCode=0 Jul 30 03:45:51.883893 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:51.883861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"6e8dff467e082a6a0a903480bec9efc0229742de270c9c4393656c8db17084fa"} Jul 30 03:45:53.561367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:53.561335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:53.562032 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:53.561381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:53.562032 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:53.561480 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:53.562032 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:53.561863 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:53.562032 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:53.561913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:53.562032 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:53.561979 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:54.102063 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:54.102013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:54.102236 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.102162 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:54.102236 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.102235 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret podName:5b45e529-005c-4897-890c-6c53db7030d3 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:26.102213989 +0000 UTC m=+66.147713189 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret") pod "global-pull-secret-syncer-ch58f" (UID: "5b45e529-005c-4897-890c-6c53db7030d3") : object "kube-system"/"original-pull-secret" not registered Jul 30 03:45:54.202411 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:54.202364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:54.202581 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:54.202453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:54.202581 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202526 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:54.202672 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202601 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 03:45:54.202672 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202619 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 03:45:54.202672 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202635 2568 projected.go:196] Error preparing data for projected volume kube-api-access-2bxjj for pod openshift-network-diagnostics/network-check-target-vdg9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:54.202672 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202607 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs podName:2df5d138-642c-4381-b83b-6ab0e3520c87 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:26.202587211 +0000 UTC m=+66.248086386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs") pod "network-metrics-daemon-8tb9g" (UID: "2df5d138-642c-4381-b83b-6ab0e3520c87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 03:45:54.202808 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:54.202698 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj podName:c684a5be-8c6c-48fd-822b-c5f5f7f5e868 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:26.202674917 +0000 UTC m=+66.248174094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bxjj" (UniqueName: "kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj") pod "network-check-target-vdg9f" (UID: "c684a5be-8c6c-48fd-822b-c5f5f7f5e868") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 03:45:55.561630 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:55.561594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:55.562091 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:55.561594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:55.562091 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:55.561728 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:55.562091 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:55.561598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:55.562091 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:55.561805 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:55.562091 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:55.561856 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:56.898290 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:56.898249 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="98d9248e1af973cb6e24abb6b056f3ef631d6df016c57c9f256649a107db4c17" exitCode=0 Jul 30 03:45:56.898290 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:56.898293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"98d9248e1af973cb6e24abb6b056f3ef631d6df016c57c9f256649a107db4c17"} Jul 30 03:45:57.561427 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:57.561381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:57.561658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:57.561434 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:57.561658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:57.561534 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:57.561658 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:57.561552 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:57.561658 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:57.561611 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:45:57.561878 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:57.561669 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:57.903938 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:57.903862 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ed8dbc-9e70-479f-b56f-988991b5c082" containerID="7a67dc8e720e36a7d508b7bf2cea4bb43d7d457d2d9d1259907edf397ceedf5e" exitCode=0 Jul 30 03:45:57.903938 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:57.903914 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerDied","Data":"7a67dc8e720e36a7d508b7bf2cea4bb43d7d457d2d9d1259907edf397ceedf5e"} Jul 30 03:45:58.909273 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:58.909011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" event={"ID":"f9ed8dbc-9e70-479f-b56f-988991b5c082","Type":"ContainerStarted","Data":"24a782bcbb365ff1c31b5ae847f8b7dbc376ab38101ef1fe3bed71da11572fb4"} Jul 30 03:45:58.928845 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:58.928797 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6vrb7" podStartSLOduration=9.176615044 podStartE2EDuration="38.928779922s" totalImagesPullingTime="29.752164878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:58.9285021 +0000 UTC m=+38.974001295" watchObservedRunningTime="2026-07-30 03:45:58.928779922 +0000 UTC m=+38.974279116" Jul 30 03:45:59.561692 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:59.561654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:45:59.561692 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:59.561681 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:45:59.561692 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:45:59.561654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:45:59.561955 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:59.561781 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:45:59.561955 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:59.561830 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:45:59.561955 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:45:59.561908 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:01.561501 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:01.561451 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:01.561501 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:01.561483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:01.561501 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:01.561483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:01.561951 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:01.561574 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:46:01.561951 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:01.561624 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:46:01.561951 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:01.561692 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:03.561800 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:03.561760 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:03.562294 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:03.561810 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:03.562294 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:03.561810 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:03.562294 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:03.561876 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:46:03.562294 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:03.562009 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:03.562294 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:03.562066 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:46:05.562181 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:05.562147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:05.562181 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:05.562205 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:05.562698 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:05.562213 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:05.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:05.562280 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:05.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:05.562345 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:46:05.562698 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:05.562436 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:46:07.562083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:07.562051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:07.562083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:07.562085 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:07.562591 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:07.562150 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:46:07.562591 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:07.562167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:07.562591 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:07.562243 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:07.562591 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:07.562294 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:46:09.561318 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:09.561275 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:09.561318 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:09.561306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:09.561796 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:09.561393 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:09.561796 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:09.561410 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8tb9g" podUID="2df5d138-642c-4381-b83b-6ab0e3520c87" Jul 30 03:46:09.561796 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:09.561521 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch58f" podUID="5b45e529-005c-4897-890c-6c53db7030d3" Jul 30 03:46:09.561796 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:09.561545 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdg9f" podUID="c684a5be-8c6c-48fd-822b-c5f5f7f5e868" Jul 30 03:46:10.298144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.298067 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-114.ec2.internal" event="NodeReady" Jul 30 03:46:10.298288 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.298232 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 03:46:10.351412 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.351378 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z5pwp"] Jul 30 03:46:10.370587 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.370555 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t8mzj"] Jul 30 03:46:10.370770 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.370679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.373116 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.373010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s29r\"" Jul 30 03:46:10.373694 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.373675 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 03:46:10.373798 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.373714 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 03:46:10.384703 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.384683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5pwp"] Jul 30 03:46:10.384803 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.384723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t8mzj"] Jul 30 03:46:10.384844 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.384799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.387131 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.387108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 03:46:10.387239 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.387173 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbcm\"" Jul 30 03:46:10.387239 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.387198 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 03:46:10.387239 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.387224 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 03:46:10.387394 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.387228 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 03:46:10.448560 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.448527 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5d5kn"] Jul 30 03:46:10.472845 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.472814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d5kn"] Jul 30 03:46:10.472988 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.472905 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.475661 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.475642 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxgj5\"" Jul 30 03:46:10.475860 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.475847 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 03:46:10.476074 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.476057 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 03:46:10.476129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.476066 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 03:46:10.513383 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.513557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f856dd4b-e520-4d1d-8704-7adfff2953c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.513557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c59794b4-1156-4a33-8e85-79e5c858079a-metrics-tls\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.513557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c59794b4-1156-4a33-8e85-79e5c858079a-config-volume\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.513557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59794b4-1156-4a33-8e85-79e5c858079a-tmp-dir\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.513557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfnk\" (UniqueName: \"kubernetes.io/projected/c59794b4-1156-4a33-8e85-79e5c858079a-kube-api-access-jqfnk\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.513720 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513569 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f856dd4b-e520-4d1d-8704-7adfff2953c5-data-volume\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.516301 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.513634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsgc\" (UniqueName: \"kubernetes.io/projected/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-api-access-9zsgc\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.516301 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.514013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f856dd4b-e520-4d1d-8704-7adfff2953c5-crio-socket\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617326 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsgc\" (UniqueName: \"kubernetes.io/projected/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-api-access-9zsgc\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617326 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f856dd4b-e520-4d1d-8704-7adfff2953c5-crio-socket\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617326 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnmd\" (UniqueName: \"kubernetes.io/projected/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-kube-api-access-qhnmd\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f856dd4b-e520-4d1d-8704-7adfff2953c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c59794b4-1156-4a33-8e85-79e5c858079a-metrics-tls\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-cert\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c59794b4-1156-4a33-8e85-79e5c858079a-config-volume\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59794b4-1156-4a33-8e85-79e5c858079a-tmp-dir\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617543 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f856dd4b-e520-4d1d-8704-7adfff2953c5-crio-socket\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfnk\" (UniqueName: \"kubernetes.io/projected/c59794b4-1156-4a33-8e85-79e5c858079a-kube-api-access-jqfnk\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.617937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f856dd4b-e520-4d1d-8704-7adfff2953c5-data-volume\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.618330 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59794b4-1156-4a33-8e85-79e5c858079a-tmp-dir\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.618330 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.617994 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.618330 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.618082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f856dd4b-e520-4d1d-8704-7adfff2953c5-data-volume\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.618330 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.618149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c59794b4-1156-4a33-8e85-79e5c858079a-config-volume\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.621738 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.621714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c59794b4-1156-4a33-8e85-79e5c858079a-metrics-tls\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.621851 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.621738 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f856dd4b-e520-4d1d-8704-7adfff2953c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.624794 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.624771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfnk\" (UniqueName: \"kubernetes.io/projected/c59794b4-1156-4a33-8e85-79e5c858079a-kube-api-access-jqfnk\") pod \"dns-default-z5pwp\" (UID: \"c59794b4-1156-4a33-8e85-79e5c858079a\") " pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.625690 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.625667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsgc\" (UniqueName: \"kubernetes.io/projected/f856dd4b-e520-4d1d-8704-7adfff2953c5-kube-api-access-9zsgc\") pod \"insights-runtime-extractor-t8mzj\" (UID: \"f856dd4b-e520-4d1d-8704-7adfff2953c5\") " pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.679725 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.679684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:10.692874 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.692838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t8mzj" Jul 30 03:46:10.718660 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.718630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-cert\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.718774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.718750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnmd\" (UniqueName: \"kubernetes.io/projected/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-kube-api-access-qhnmd\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.721216 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.721195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-cert\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.726588 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.726564 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnmd\" (UniqueName: \"kubernetes.io/projected/0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90-kube-api-access-qhnmd\") pod \"ingress-canary-5d5kn\" (UID: \"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90\") " pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.781316 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.781277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d5kn" Jul 30 03:46:10.877095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.876888 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5pwp"] Jul 30 03:46:10.881944 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:10.881903 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59794b4_1156_4a33_8e85_79e5c858079a.slice/crio-af8a86828295b6adbf62d032fa2c045051d2320c4f26702b6d81cfcc1d1deeff WatchSource:0}: Error finding container af8a86828295b6adbf62d032fa2c045051d2320c4f26702b6d81cfcc1d1deeff: Status 404 returned error can't find the container with id af8a86828295b6adbf62d032fa2c045051d2320c4f26702b6d81cfcc1d1deeff Jul 30 03:46:10.896823 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.896798 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t8mzj"] Jul 30 03:46:10.901033 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:10.901005 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf856dd4b_e520_4d1d_8704_7adfff2953c5.slice/crio-0bbadfd372fe41883873f708bf0db05884aedf8bc68534113276d54dab3b10c1 WatchSource:0}: Error finding container 0bbadfd372fe41883873f708bf0db05884aedf8bc68534113276d54dab3b10c1: Status 404 returned error can't find the container with id 0bbadfd372fe41883873f708bf0db05884aedf8bc68534113276d54dab3b10c1 Jul 30 03:46:10.937654 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.937617 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8mzj" event={"ID":"f856dd4b-e520-4d1d-8704-7adfff2953c5","Type":"ContainerStarted","Data":"0bbadfd372fe41883873f708bf0db05884aedf8bc68534113276d54dab3b10c1"} Jul 30 03:46:10.938615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.938591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5pwp" event={"ID":"c59794b4-1156-4a33-8e85-79e5c858079a","Type":"ContainerStarted","Data":"af8a86828295b6adbf62d032fa2c045051d2320c4f26702b6d81cfcc1d1deeff"} Jul 30 03:46:10.943659 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:10.943636 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d5kn"] Jul 30 03:46:10.948489 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:10.948449 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5ad207_eeaf_4bf2_b21d_0dd9b1467f90.slice/crio-f8ad1fe22b1ce1991aba2d2b9482e004b521c6c6db6eec55fda0cf546112fd21 WatchSource:0}: Error finding container f8ad1fe22b1ce1991aba2d2b9482e004b521c6c6db6eec55fda0cf546112fd21: Status 404 returned error can't find the container with id f8ad1fe22b1ce1991aba2d2b9482e004b521c6c6db6eec55fda0cf546112fd21 Jul 30 03:46:11.561650 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.561612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:11.562031 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.561618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:11.562367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.561630 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565257 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzw94\"" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565370 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565375 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 03:46:11.565611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.565542 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmgxg\"" Jul 30 03:46:11.949291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:11.949248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d5kn" event={"ID":"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90","Type":"ContainerStarted","Data":"f8ad1fe22b1ce1991aba2d2b9482e004b521c6c6db6eec55fda0cf546112fd21"} Jul 30 03:46:17.965424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.965385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8mzj" event={"ID":"f856dd4b-e520-4d1d-8704-7adfff2953c5","Type":"ContainerStarted","Data":"27894dea7deff9d207cd813671eae9e7e9f59b631c08f943ced590e5c53991bf"} Jul 30 03:46:17.967430 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.967404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5pwp" event={"ID":"c59794b4-1156-4a33-8e85-79e5c858079a","Type":"ContainerStarted","Data":"638091f34bddef7c493f51fb488c7ce54b0ce21bcf18c786fa266f6a44710c54"} Jul 30 03:46:17.967583 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.967448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5pwp" event={"ID":"c59794b4-1156-4a33-8e85-79e5c858079a","Type":"ContainerStarted","Data":"8ca6320c4aabd0157c01b782e0e41d9f72138c6f692475616ac79ba8e648567d"} Jul 30 03:46:17.967583 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.967502 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:17.968845 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.968822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d5kn" event={"ID":"0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90","Type":"ContainerStarted","Data":"b9fdbcaa2a34ec89d73990763f0c9f306d8ed060354cedf41096689fda7aab14"} Jul 30 03:46:17.982747 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.982671 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z5pwp" podStartSLOduration=1.8831391910000002 podStartE2EDuration="7.98265814s" totalImagesPullingTime="6.099518949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:17.981868892 +0000 UTC m=+58.027368101" watchObservedRunningTime="2026-07-30 03:46:17.98265814 +0000 UTC m=+58.028157406" Jul 30 03:46:17.995313 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:17.995272 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5d5kn" podStartSLOduration=1.554380249 podStartE2EDuration="7.995258748s" totalImagesPullingTime="6.440878499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:17.994611068 +0000 UTC m=+58.040110263" watchObservedRunningTime="2026-07-30 03:46:17.995258748 +0000 UTC m=+58.040757978" Jul 30 03:46:19.130087 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.130050 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s"] Jul 30 03:46:19.132188 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.132160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:19.134658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.134630 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 30 03:46:19.134766 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.134668 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7w5j8\"" Jul 30 03:46:19.139551 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.139525 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s"] Jul 30 03:46:19.199219 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.199181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7lf4s\" (UID: \"f4cf6d70-21b0-4319-b424-07e4f8390b7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:19.300045 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.300006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7lf4s\" (UID: \"f4cf6d70-21b0-4319-b424-07e4f8390b7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:19.300213 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:19.300156 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 30 03:46:19.300255 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:19.300224 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates podName:f4cf6d70-21b0-4319-b424-07e4f8390b7b nodeName:}" failed. No retries permitted until 2026-07-30 03:46:19.800206691 +0000 UTC m=+59.845705865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" (UID: "f4cf6d70-21b0-4319-b424-07e4f8390b7b") : secret "prometheus-operator-admission-webhook-tls" not found Jul 30 03:46:19.803603 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:19.803558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7lf4s\" (UID: \"f4cf6d70-21b0-4319-b424-07e4f8390b7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:19.803799 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:19.803717 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 30 03:46:19.803799 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:19.803797 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates podName:f4cf6d70-21b0-4319-b424-07e4f8390b7b nodeName:}" failed. No retries permitted until 2026-07-30 03:46:20.803774305 +0000 UTC m=+60.849273494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" (UID: "f4cf6d70-21b0-4319-b424-07e4f8390b7b") : secret "prometheus-operator-admission-webhook-tls" not found Jul 30 03:46:20.814549 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:20.814502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7lf4s\" (UID: \"f4cf6d70-21b0-4319-b424-07e4f8390b7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:20.817236 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:20.817206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4cf6d70-21b0-4319-b424-07e4f8390b7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7lf4s\" (UID: \"f4cf6d70-21b0-4319-b424-07e4f8390b7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:20.889604 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:20.889577 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpn5x" Jul 30 03:46:20.945500 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:20.945452 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7w5j8\"" Jul 30 03:46:20.954263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:20.954230 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:21.611601 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:21.611574 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s"] Jul 30 03:46:21.614809 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:21.614784 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf6d70_21b0_4319_b424_07e4f8390b7b.slice/crio-df6c8c3d5cd2b5c57f83344a4fc70d3579ee61d86bd74d2f6e8afa9a57c91307 WatchSource:0}: Error finding container df6c8c3d5cd2b5c57f83344a4fc70d3579ee61d86bd74d2f6e8afa9a57c91307: Status 404 returned error can't find the container with id df6c8c3d5cd2b5c57f83344a4fc70d3579ee61d86bd74d2f6e8afa9a57c91307 Jul 30 03:46:21.982028 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:21.981984 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8mzj" event={"ID":"f856dd4b-e520-4d1d-8704-7adfff2953c5","Type":"ContainerStarted","Data":"1fdc2496a535be85a4f3c8ab06da9e8c745c008f7c889b21eae91f8e28a2e35d"} Jul 30 03:46:21.983647 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:21.983618 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" event={"ID":"f4cf6d70-21b0-4319-b424-07e4f8390b7b","Type":"ContainerStarted","Data":"df6c8c3d5cd2b5c57f83344a4fc70d3579ee61d86bd74d2f6e8afa9a57c91307"} Jul 30 03:46:22.988638 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:22.988599 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" event={"ID":"f4cf6d70-21b0-4319-b424-07e4f8390b7b","Type":"ContainerStarted","Data":"d75bc5a30403fa36714efbe672edb6ba569f296a1eeda5f738548e5fc062fee4"} Jul 30 03:46:22.989083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:22.988874 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:22.993933 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:22.993900 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" Jul 30 03:46:23.003614 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.003570 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7lf4s" podStartSLOduration=2.84023562 podStartE2EDuration="4.003555397s" totalImagesPullingTime="1.163319777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:23.002214276 +0000 UTC m=+63.047713471" watchObservedRunningTime="2026-07-30 03:46:23.003555397 +0000 UTC m=+63.049054591" Jul 30 03:46:23.189241 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.189209 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v47mr"] Jul 30 03:46:23.191295 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.191271 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.193568 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193540 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 03:46:23.193655 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193550 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 03:46:23.193883 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193861 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jf49p\"" Jul 30 03:46:23.193957 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193925 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 03:46:23.193957 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193867 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 03:46:23.194045 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.193867 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 03:46:23.200035 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.200015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v47mr"] Jul 30 03:46:23.235717 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.235689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.235866 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.235726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.235866 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.235770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfvv\" (UniqueName: \"kubernetes.io/projected/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-kube-api-access-qqfvv\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.235866 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.235801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.336730 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.336689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.336896 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.336744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.336896 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.336795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfvv\" (UniqueName: \"kubernetes.io/projected/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-kube-api-access-qqfvv\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.336896 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.336823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.336896 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:23.336859 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 30 03:46:23.337107 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:23.336936 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls podName:5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:23.836916315 +0000 UTC m=+63.882415488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-v47mr" (UID: "5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2") : secret "prometheus-operator-tls" not found Jul 30 03:46:23.337652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.337631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.339682 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.339660 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.345526 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.345498 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfvv\" (UniqueName: \"kubernetes.io/projected/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-kube-api-access-qqfvv\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.840302 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.840264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:23.842951 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:23.842912 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v47mr\" (UID: \"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:24.102238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:24.102160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" Jul 30 03:46:25.165249 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:25.165218 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v47mr"] Jul 30 03:46:25.168795 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:25.168766 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea1e26c_7f9c_48d8_a6ba_276c4d2d02d2.slice/crio-4f77df3eb1ef0738ca1469fd6da8e503c994c32e3fbfb288f69b12d759c5570b WatchSource:0}: Error finding container 4f77df3eb1ef0738ca1469fd6da8e503c994c32e3fbfb288f69b12d759c5570b: Status 404 returned error can't find the container with id 4f77df3eb1ef0738ca1469fd6da8e503c994c32e3fbfb288f69b12d759c5570b Jul 30 03:46:26.002752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.002699 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8mzj" event={"ID":"f856dd4b-e520-4d1d-8704-7adfff2953c5","Type":"ContainerStarted","Data":"95d1d81c1872aca662838ff97fea55fe57a8460a083c8e897ad5643b99a88929"} Jul 30 03:46:26.004115 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.004090 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" event={"ID":"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2","Type":"ContainerStarted","Data":"4f77df3eb1ef0738ca1469fd6da8e503c994c32e3fbfb288f69b12d759c5570b"} Jul 30 03:46:26.019692 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.019640 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t8mzj" podStartSLOduration=2.015059686 podStartE2EDuration="16.019618668s" totalImagesPullingTime="14.004558982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:46:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:26.018947893 +0000 UTC m=+66.064447089" watchObservedRunningTime="2026-07-30 03:46:26.019618668 +0000 UTC m=+66.065117927" Jul 30 03:46:26.159734 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.159683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:26.162188 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.162162 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 03:46:26.172787 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.172763 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b45e529-005c-4897-890c-6c53db7030d3-original-pull-secret\") pod \"global-pull-secret-syncer-ch58f\" (UID: \"5b45e529-005c-4897-890c-6c53db7030d3\") " pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:26.260319 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.260226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:26.260319 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.260297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:26.262576 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.262547 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 03:46:26.262710 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.262551 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 03:46:26.272500 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.272450 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 03:46:26.273241 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.273217 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df5d138-642c-4381-b83b-6ab0e3520c87-metrics-certs\") pod \"network-metrics-daemon-8tb9g\" (UID: \"2df5d138-642c-4381-b83b-6ab0e3520c87\") " pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:26.277068 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.277045 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzw94\"" Jul 30 03:46:26.283412 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.283390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxjj\" (UniqueName: \"kubernetes.io/projected/c684a5be-8c6c-48fd-822b-c5f5f7f5e868-kube-api-access-2bxjj\") pod \"network-check-target-vdg9f\" (UID: \"c684a5be-8c6c-48fd-822b-c5f5f7f5e868\") " pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:26.286073 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.286049 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8tb9g" Jul 30 03:46:26.286183 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.286084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch58f" Jul 30 03:46:26.294927 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.294896 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmgxg\"" Jul 30 03:46:26.303307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.303281 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:26.444438 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.443084 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8tb9g"] Jul 30 03:46:26.465372 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.465341 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch58f"] Jul 30 03:46:26.484454 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:26.484426 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdg9f"] Jul 30 03:46:26.709998 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:26.709955 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc684a5be_8c6c_48fd_822b_c5f5f7f5e868.slice/crio-8323bfd1f9738c655b5bf8ebdb14585518430f6f1c8276fd9d5d960ff270682e WatchSource:0}: Error finding container 8323bfd1f9738c655b5bf8ebdb14585518430f6f1c8276fd9d5d960ff270682e: Status 404 returned error can't find the container with id 8323bfd1f9738c655b5bf8ebdb14585518430f6f1c8276fd9d5d960ff270682e Jul 30 03:46:26.710365 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:26.710342 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b45e529_005c_4897_890c_6c53db7030d3.slice/crio-0094d8263bc848a646fc5c281bc5c15d63b493bbec94b060f0a5ad7557e721e9 WatchSource:0}: Error finding container 0094d8263bc848a646fc5c281bc5c15d63b493bbec94b060f0a5ad7557e721e9: Status 404 returned error can't find the container with id 0094d8263bc848a646fc5c281bc5c15d63b493bbec94b060f0a5ad7557e721e9 Jul 30 03:46:26.710995 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:26.710883 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df5d138_642c_4381_b83b_6ab0e3520c87.slice/crio-f30d44492dc7829e2ac36b327eee6ab5338c10be189f65168260ab0cabceb98b WatchSource:0}: Error finding container f30d44492dc7829e2ac36b327eee6ab5338c10be189f65168260ab0cabceb98b: Status 404 returned error can't find the container with id f30d44492dc7829e2ac36b327eee6ab5338c10be189f65168260ab0cabceb98b Jul 30 03:46:27.008813 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.008716 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdg9f" event={"ID":"c684a5be-8c6c-48fd-822b-c5f5f7f5e868","Type":"ContainerStarted","Data":"8323bfd1f9738c655b5bf8ebdb14585518430f6f1c8276fd9d5d960ff270682e"} Jul 30 03:46:27.009936 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.009905 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tb9g" event={"ID":"2df5d138-642c-4381-b83b-6ab0e3520c87","Type":"ContainerStarted","Data":"f30d44492dc7829e2ac36b327eee6ab5338c10be189f65168260ab0cabceb98b"} Jul 30 03:46:27.011003 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.010979 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch58f" event={"ID":"5b45e529-005c-4897-890c-6c53db7030d3","Type":"ContainerStarted","Data":"0094d8263bc848a646fc5c281bc5c15d63b493bbec94b060f0a5ad7557e721e9"} Jul 30 03:46:27.012767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.012742 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" event={"ID":"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2","Type":"ContainerStarted","Data":"aeb67645e2c4a93487d6a2a1d1dd3c03e14cb9cea38fe5ddb6dc75e3fdeceba4"} Jul 30 03:46:27.012858 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.012794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" event={"ID":"5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2","Type":"ContainerStarted","Data":"d9da5f2921b382141be1e1e7812dcbee6bd923ec343951fe07467ee1c9029c1a"} Jul 30 03:46:27.030404 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.030333 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-v47mr" podStartSLOduration=2.464169902 podStartE2EDuration="4.030315824s" totalImagesPullingTime="1.566145922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:27.02895862 +0000 UTC m=+67.074457815" watchObservedRunningTime="2026-07-30 03:46:27.030315824 +0000 UTC m=+67.075815018" Jul 30 03:46:27.974594 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:27.974509 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5pwp" Jul 30 03:46:28.021184 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.021140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tb9g" event={"ID":"2df5d138-642c-4381-b83b-6ab0e3520c87","Type":"ContainerStarted","Data":"9043a711d4a984959686748443e8865a968f0d992a5caf14e3ac23ab5ec0eb69"} Jul 30 03:46:28.021361 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.021206 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8tb9g" event={"ID":"2df5d138-642c-4381-b83b-6ab0e3520c87","Type":"ContainerStarted","Data":"3f1e0177d2d84cffcbe714e11e661165ceb2532db50c65e146b9537f18916ad1"} Jul 30 03:46:28.045019 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.044306 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8tb9g" podStartSLOduration=67.084248383 podStartE2EDuration="1m8.044281831s" totalImagesPullingTime="960.033448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:28.043351599 +0000 UTC m=+68.088850806" watchObservedRunningTime="2026-07-30 03:46:28.044281831 +0000 UTC m=+68.089781026" Jul 30 03:46:28.532629 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.532592 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj"] Jul 30 03:46:28.535336 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.535306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.537967 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.537945 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wtlf8\"" Jul 30 03:46:28.538424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.538404 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 30 03:46:28.538653 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.538637 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 30 03:46:28.545624 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.545603 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj"] Jul 30 03:46:28.548973 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.548949 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r"] Jul 30 03:46:28.553021 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.552992 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.557592 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.557572 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-px6sb"] Jul 30 03:46:28.558155 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.558128 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 03:46:28.558400 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.558381 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 03:46:28.558781 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.558763 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 03:46:28.559256 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.559223 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vh5jj\"" Jul 30 03:46:28.559773 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.559751 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.561949 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.561926 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 03:46:28.561949 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.561940 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 03:46:28.562111 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.562066 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fg6jg\"" Jul 30 03:46:28.566174 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.566152 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 03:46:28.568931 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.568910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r"] Jul 30 03:46:28.576759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c300a760-a61e-4454-a5f2-98463d5387fe-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.576872 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576784 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vd6\" (UniqueName: \"kubernetes.io/projected/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-api-access-p8vd6\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.576872 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576818 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.576872 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.577003 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.577003 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwl5\" (UniqueName: \"kubernetes.io/projected/c300a760-a61e-4454-a5f2-98463d5387fe-kube-api-access-txwl5\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.577003 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.577003 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.576989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.577193 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.577015 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.577193 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.577100 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.678018 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.677968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-root\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678197 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.678197 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c300a760-a61e-4454-a5f2-98463d5387fe-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.678197 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.678367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwl5\" (UniqueName: \"kubernetes.io/projected/c300a760-a61e-4454-a5f2-98463d5387fe-kube-api-access-txwl5\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.678367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-metrics-client-ca\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.678367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vd6\" (UniqueName: \"kubernetes.io/projected/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-api-access-p8vd6\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-accelerators-collector-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-sys\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-textfile\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldj58\" (UniqueName: \"kubernetes.io/projected/230b0d8b-67b2-4e76-9fe9-858119ee618e-kube-api-access-ldj58\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.678634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-wtmp\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.679051 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.679051 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.678964 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c300a760-a61e-4454-a5f2-98463d5387fe-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.679312 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:28.679074 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 30 03:46:28.679312 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:28.679133 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls podName:400d1899-9cd9-4c9a-96d8-1cff2e417f4e nodeName:}" failed. No retries permitted until 2026-07-30 03:46:29.179114023 +0000 UTC m=+69.224613195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-6z88r" (UID: "400d1899-9cd9-4c9a-96d8-1cff2e417f4e") : secret "kube-state-metrics-tls" not found Jul 30 03:46:28.679312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.679254 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.679312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.679263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.679556 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.679440 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.681430 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.681407 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.682089 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.682067 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c300a760-a61e-4454-a5f2-98463d5387fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.682227 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.682191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.692690 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.692487 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vd6\" (UniqueName: \"kubernetes.io/projected/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-api-access-p8vd6\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:28.693156 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.693117 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwl5\" (UniqueName: \"kubernetes.io/projected/c300a760-a61e-4454-a5f2-98463d5387fe-kube-api-access-txwl5\") pod \"openshift-state-metrics-b8f49f5c6-4sjgj\" (UID: \"c300a760-a61e-4454-a5f2-98463d5387fe\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:28.779067 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-metrics-client-ca\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-accelerators-collector-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-sys\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-textfile\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779169 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldj58\" (UniqueName: \"kubernetes.io/projected/230b0d8b-67b2-4e76-9fe9-858119ee618e-kube-api-access-ldj58\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-wtmp\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779240 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-root\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779657 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779305 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-root\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.779984 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.779957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-textfile\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.780129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.780027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-sys\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.780306 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.780284 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-accelerators-collector-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.780393 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:28.780371 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 03:46:28.780455 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:28.780422 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls podName:230b0d8b-67b2-4e76-9fe9-858119ee618e nodeName:}" failed. No retries permitted until 2026-07-30 03:46:29.280408208 +0000 UTC m=+69.325907381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls") pod "node-exporter-px6sb" (UID: "230b0d8b-67b2-4e76-9fe9-858119ee618e") : secret "node-exporter-tls" not found Jul 30 03:46:28.780543 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.780486 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/230b0d8b-67b2-4e76-9fe9-858119ee618e-metrics-client-ca\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.780605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.780554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-wtmp\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.782172 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.782147 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.790608 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.790555 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldj58\" (UniqueName: \"kubernetes.io/projected/230b0d8b-67b2-4e76-9fe9-858119ee618e-kube-api-access-ldj58\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:28.849398 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:28.849357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" Jul 30 03:46:29.183319 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.183279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:29.186593 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.186557 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d1899-9cd9-4c9a-96d8-1cff2e417f4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6z88r\" (UID: \"400d1899-9cd9-4c9a-96d8-1cff2e417f4e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:29.284742 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.284695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:29.287778 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.287743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/230b0d8b-67b2-4e76-9fe9-858119ee618e-node-exporter-tls\") pod \"node-exporter-px6sb\" (UID: \"230b0d8b-67b2-4e76-9fe9-858119ee618e\") " pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:29.470774 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.470685 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" Jul 30 03:46:29.478036 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.478011 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px6sb" Jul 30 03:46:29.610280 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.610242 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:46:29.631057 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.631030 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.632974 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.632943 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:46:29.633834 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.633810 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 03:46:29.633960 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.633871 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 03:46:29.634143 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634123 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 03:46:29.634244 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634165 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 03:46:29.634342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634322 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qn7td\"" Jul 30 03:46:29.634418 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 03:46:29.634638 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634619 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 03:46:29.634702 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634686 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 03:46:29.634772 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.634625 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 03:46:29.634925 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.633811 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 03:46:29.688645 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.688826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.688826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.688826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.688826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.688826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688818 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688903 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj49\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.688988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.689039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.689012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790523 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj49\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:29.790665 2568 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:46:29.790742 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls podName:5893ca92-23fb-45a3-8079-3fe7f3768280 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:30.290717856 +0000 UTC m=+70.336217031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280") : secret "alertmanager-main-tls" not found Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.790865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.791710 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.790878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.791961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.791937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.792385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.792363 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.793528 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.793503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.794392 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.793983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.794392 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.794240 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.795395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.795371 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.795795 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.795771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.795990 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.795969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.796070 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.795987 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.796180 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.796156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.796315 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.796295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:29.799395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:29.799374 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj49\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:30.295441 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.295390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:30.298301 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.298268 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:30.530493 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.529020 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bd7d888b8-5xgx4"] Jul 30 03:46:30.543627 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.543597 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:46:30.553044 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.552965 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bd7d888b8-5xgx4"] Jul 30 03:46:30.553195 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.553087 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.555376 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555350 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 30 03:46:30.555634 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555614 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6pgzd\"" Jul 30 03:46:30.555735 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555670 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c3dcvcemi8a72\"" Jul 30 03:46:30.555735 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555683 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 30 03:46:30.555735 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555446 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 30 03:46:30.555897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 30 03:46:30.555897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.555501 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 30 03:46:30.598011 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.597965 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598011 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598059 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-grpc-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcdm\" (UniqueName: \"kubernetes.io/projected/dd9ece92-d257-49c6-be38-4147864a1b66-kube-api-access-sjcdm\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9ece92-d257-49c6-be38-4147864a1b66-metrics-client-ca\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.598499 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.598279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699458 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9ece92-d257-49c6-be38-4147864a1b66-metrics-client-ca\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-grpc-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcdm\" (UniqueName: \"kubernetes.io/projected/dd9ece92-d257-49c6-be38-4147864a1b66-kube-api-access-sjcdm\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.699961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.699669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.700253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.700226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9ece92-d257-49c6-be38-4147864a1b66-metrics-client-ca\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.703671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.703640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-grpc-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.704132 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.704104 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.704237 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.703792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-tls\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.704237 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.704095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.704647 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.704622 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.705351 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.705023 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9ece92-d257-49c6-be38-4147864a1b66-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.707553 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.707454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcdm\" (UniqueName: \"kubernetes.io/projected/dd9ece92-d257-49c6-be38-4147864a1b66-kube-api-access-sjcdm\") pod \"thanos-querier-bd7d888b8-5xgx4\" (UID: \"dd9ece92-d257-49c6-be38-4147864a1b66\") " pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:30.866018 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:30.865922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:31.645600 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:31.645513 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj"] Jul 30 03:46:31.649847 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:31.649807 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc300a760_a61e_4454_a5f2_98463d5387fe.slice/crio-6648e99795b402c1a305e08d4e89d8987d12e56e2a9d0126c6ca75e53cf93ce9 WatchSource:0}: Error finding container 6648e99795b402c1a305e08d4e89d8987d12e56e2a9d0126c6ca75e53cf93ce9: Status 404 returned error can't find the container with id 6648e99795b402c1a305e08d4e89d8987d12e56e2a9d0126c6ca75e53cf93ce9 Jul 30 03:46:31.669345 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:31.669177 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:46:31.684238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:31.684161 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r"] Jul 30 03:46:31.687838 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:31.687800 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d1899_9cd9_4c9a_96d8_1cff2e417f4e.slice/crio-e24ad45c04159415af6e7f7f4afef88b7bb8564454f78c917ff0ff1669616e78 WatchSource:0}: Error finding container e24ad45c04159415af6e7f7f4afef88b7bb8564454f78c917ff0ff1669616e78: Status 404 returned error can't find the container with id e24ad45c04159415af6e7f7f4afef88b7bb8564454f78c917ff0ff1669616e78 Jul 30 03:46:31.700424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:31.700305 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bd7d888b8-5xgx4"] Jul 30 03:46:31.705821 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:31.705786 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9ece92_d257_49c6_be38_4147864a1b66.slice/crio-1b0fb77668fdde08f030d9d7068f7a721749be20a38c2ae8f390c99d5a886c34 WatchSource:0}: Error finding container 1b0fb77668fdde08f030d9d7068f7a721749be20a38c2ae8f390c99d5a886c34: Status 404 returned error can't find the container with id 1b0fb77668fdde08f030d9d7068f7a721749be20a38c2ae8f390c99d5a886c34 Jul 30 03:46:32.041737 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.041645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"1b0fb77668fdde08f030d9d7068f7a721749be20a38c2ae8f390c99d5a886c34"} Jul 30 03:46:32.043079 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.043047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px6sb" event={"ID":"230b0d8b-67b2-4e76-9fe9-858119ee618e","Type":"ContainerStarted","Data":"7f8258ecdff02775ce21abe5088d107712fc0622aa406bf885b2b9ea276c8683"} Jul 30 03:46:32.044503 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.044450 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"3d39aaf539c3f7a088b73d33c3a41d19867f84ea218da63b6fb02beb504753dd"} Jul 30 03:46:32.046065 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.046038 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdg9f" event={"ID":"c684a5be-8c6c-48fd-822b-c5f5f7f5e868","Type":"ContainerStarted","Data":"8f3c8fb9300fb016aeb7347be3b4fc3695e76718397516ff60a4003d3c9d161e"} Jul 30 03:46:32.046162 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.046116 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:46:32.047563 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.047534 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" event={"ID":"400d1899-9cd9-4c9a-96d8-1cff2e417f4e","Type":"ContainerStarted","Data":"e24ad45c04159415af6e7f7f4afef88b7bb8564454f78c917ff0ff1669616e78"} Jul 30 03:46:32.049524 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.049496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" event={"ID":"c300a760-a61e-4454-a5f2-98463d5387fe","Type":"ContainerStarted","Data":"5710a563eec3e53834245a584aa9f9eccd710fc6d5b161574ae76dfbbcceb3b0"} Jul 30 03:46:32.049610 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.049563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" event={"ID":"c300a760-a61e-4454-a5f2-98463d5387fe","Type":"ContainerStarted","Data":"bbfb63b489d1b0e6a067689399da9d4139db9dd6d8aa867d337e5ac2e5d6fc40"} Jul 30 03:46:32.049610 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.049592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" event={"ID":"c300a760-a61e-4454-a5f2-98463d5387fe","Type":"ContainerStarted","Data":"6648e99795b402c1a305e08d4e89d8987d12e56e2a9d0126c6ca75e53cf93ce9"} Jul 30 03:46:32.051049 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.051020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch58f" event={"ID":"5b45e529-005c-4897-890c-6c53db7030d3","Type":"ContainerStarted","Data":"8d76876fb5025d4749ff2f3e710b80eec5e2764d1058c8094217881892648051"} Jul 30 03:46:32.062963 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.062814 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdg9f" podStartSLOduration=67.315303513 podStartE2EDuration="1m12.06279629s" totalImagesPullingTime="4.747492777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:32.061929201 +0000 UTC m=+72.107428420" watchObservedRunningTime="2026-07-30 03:46:32.06279629 +0000 UTC m=+72.108295490" Jul 30 03:46:32.077115 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.077066 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ch58f" podStartSLOduration=67.315170536 podStartE2EDuration="1m12.077046605s" totalImagesPullingTime="4.761876069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:32.075813706 +0000 UTC m=+72.121312902" watchObservedRunningTime="2026-07-30 03:46:32.077046605 +0000 UTC m=+72.122545779" Jul 30 03:46:32.849402 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.849315 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-684dd5868f-xm6zc"] Jul 30 03:46:32.852761 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.852082 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.854865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.854629 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 30 03:46:32.856365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.855553 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 30 03:46:32.856365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.855783 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6np7f\"" Jul 30 03:46:32.856365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.855856 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 30 03:46:32.856365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.856124 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 30 03:46:32.856365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.856175 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1sts0r9jtl6nb\"" Jul 30 03:46:32.863885 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.863860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684dd5868f-xm6zc"] Jul 30 03:46:32.921764 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.921764 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55brf\" (UniqueName: \"kubernetes.io/projected/7db0b0f8-28b1-49c5-9772-ece2703fc684-kube-api-access-55brf\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.922015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-client-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.922015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-tls\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.922015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-metrics-server-audit-profiles\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.922015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921929 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-client-certs\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:32.922015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:32.921953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7db0b0f8-28b1-49c5-9772-ece2703fc684-audit-log\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-client-certs\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7db0b0f8-28b1-49c5-9772-ece2703fc684-audit-log\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55brf\" (UniqueName: \"kubernetes.io/projected/7db0b0f8-28b1-49c5-9772-ece2703fc684-kube-api-access-55brf\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-client-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-tls\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.022772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-metrics-server-audit-profiles\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.024539 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.023825 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-metrics-server-audit-profiles\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.025102 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.024648 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7db0b0f8-28b1-49c5-9772-ece2703fc684-audit-log\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.025102 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.024988 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db0b0f8-28b1-49c5-9772-ece2703fc684-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.028001 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.027951 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-tls\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.028590 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.028550 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-secret-metrics-server-client-certs\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.028959 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.028936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7db0b0f8-28b1-49c5-9772-ece2703fc684-client-ca-bundle\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.032543 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.032412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55brf\" (UniqueName: \"kubernetes.io/projected/7db0b0f8-28b1-49c5-9772-ece2703fc684-kube-api-access-55brf\") pod \"metrics-server-684dd5868f-xm6zc\" (UID: \"7db0b0f8-28b1-49c5-9772-ece2703fc684\") " pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.058624 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.058589 2568 generic.go:356] "Generic (PLEG): container finished" podID="230b0d8b-67b2-4e76-9fe9-858119ee618e" containerID="064b795dcec8df93f3e1fefccf42d16a275dd981ef8be55be7d4726866238d34" exitCode=0 Jul 30 03:46:33.059189 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.059162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px6sb" event={"ID":"230b0d8b-67b2-4e76-9fe9-858119ee618e","Type":"ContainerDied","Data":"064b795dcec8df93f3e1fefccf42d16a275dd981ef8be55be7d4726866238d34"} Jul 30 03:46:33.165856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.165829 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:33.315446 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.315410 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p"] Jul 30 03:46:33.318083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.318061 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.320341 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.320303 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 30 03:46:33.320571 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.320557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zsmlt\"" Jul 30 03:46:33.328377 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.328317 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p"] Jul 30 03:46:33.427120 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.427042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nz5\" (UniqueName: \"kubernetes.io/projected/d75da48f-0d01-4a6f-96e5-fd01b7518337-kube-api-access-f5nz5\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.427261 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.427135 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d75da48f-0d01-4a6f-96e5-fd01b7518337-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.528283 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.528243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d75da48f-0d01-4a6f-96e5-fd01b7518337-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.528453 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.528301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nz5\" (UniqueName: \"kubernetes.io/projected/d75da48f-0d01-4a6f-96e5-fd01b7518337-kube-api-access-f5nz5\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.531288 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.531255 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d75da48f-0d01-4a6f-96e5-fd01b7518337-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.536383 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.536357 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nz5\" (UniqueName: \"kubernetes.io/projected/d75da48f-0d01-4a6f-96e5-fd01b7518337-kube-api-access-f5nz5\") pod \"monitoring-plugin-68b7fc9574-n5p2p\" (UID: \"d75da48f-0d01-4a6f-96e5-fd01b7518337\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.630845 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.630808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:33.727126 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.727040 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9768d7845-5f4xm"] Jul 30 03:46:33.729729 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.729709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.732370 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 03:46:33.732515 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 03:46:33.733367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732779 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 03:46:33.733367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732781 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 03:46:33.733367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732836 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 03:46:33.733367 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.732882 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xqj4w\"" Jul 30 03:46:33.744052 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.744029 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 03:46:33.744052 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.744046 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9768d7845-5f4xm"] Jul 30 03:46:33.830911 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.830877 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831088 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.830923 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-serving-certs-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831088 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.830959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-federate-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831088 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.831040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctp9\" (UniqueName: \"kubernetes.io/projected/1e90afbe-e730-42f7-9f25-7d203667789e-kube-api-access-wctp9\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831088 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.831076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831272 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.831120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831272 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.831150 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.831272 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.831181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-metrics-client-ca\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932279 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-metrics-client-ca\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932387 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-serving-certs-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-federate-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctp9\" (UniqueName: \"kubernetes.io/projected/1e90afbe-e730-42f7-9f25-7d203667789e-kube-api-access-wctp9\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.932752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.932531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.933752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.933725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.934843 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.934273 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-serving-certs-ca-bundle\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.934843 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.934805 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e90afbe-e730-42f7-9f25-7d203667789e-metrics-client-ca\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.935540 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.935507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.935869 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.935844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-secret-telemeter-client\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.935961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.935940 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-telemeter-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.936303 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.936279 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e90afbe-e730-42f7-9f25-7d203667789e-federate-client-tls\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:33.943399 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:33.943371 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctp9\" (UniqueName: \"kubernetes.io/projected/1e90afbe-e730-42f7-9f25-7d203667789e-kube-api-access-wctp9\") pod \"telemeter-client-9768d7845-5f4xm\" (UID: \"1e90afbe-e730-42f7-9f25-7d203667789e\") " pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:34.045200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.045109 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" Jul 30 03:46:34.285491 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.284910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684dd5868f-xm6zc"] Jul 30 03:46:34.314610 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.312574 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9768d7845-5f4xm"] Jul 30 03:46:34.335184 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.335157 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p"] Jul 30 03:46:34.347901 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:34.347871 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75da48f_0d01_4a6f_96e5_fd01b7518337.slice/crio-4af4032bdc226a90eb5f8ac658e46b9052ad42edfa1574df6d0324041013a7b9 WatchSource:0}: Error finding container 4af4032bdc226a90eb5f8ac658e46b9052ad42edfa1574df6d0324041013a7b9: Status 404 returned error can't find the container with id 4af4032bdc226a90eb5f8ac658e46b9052ad42edfa1574df6d0324041013a7b9 Jul 30 03:46:34.768619 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.768581 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:46:34.773164 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.773141 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.776422 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776398 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 03:46:34.776651 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776629 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 03:46:34.776732 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 03:46:34.776732 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776726 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 03:46:34.776828 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776582 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 03:46:34.776828 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776530 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 03:46:34.776937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776586 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 03:46:34.776937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776400 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 03:46:34.776937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776636 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 03:46:34.776937 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.776584 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 03:46:34.777113 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.777018 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aln0ul09ge2m3\"" Jul 30 03:46:34.777160 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.777126 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 03:46:34.777552 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.777528 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lbnhs\"" Jul 30 03:46:34.777807 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.777789 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 03:46:34.781856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.781830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 03:46:34.785590 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.785101 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 03:46:34.788544 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.788521 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:46:34.841349 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841349 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841373 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841451 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42xj\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841621 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.841847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841894 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841931 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.841958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.842004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.842043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.842068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.842090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.842305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.842117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943249 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943249 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42xj\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943387 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.943759 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.943739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944419 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944629 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.944678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.945196 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.945598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.946390 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.946101 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.948992 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.948967 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.949406 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.949360 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.949558 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.949422 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.950058 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.950002 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.951490 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.950265 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.951490 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.951100 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.952823 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.952798 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.953712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.953223 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.953712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.953512 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.953712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.953630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.953712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.953674 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.954207 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.954166 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.954295 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.954264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.954812 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.954773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.955633 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.955613 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:34.957637 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:34.957617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42xj\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj\") pod \"prometheus-k8s-0\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:35.075579 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.074764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" event={"ID":"400d1899-9cd9-4c9a-96d8-1cff2e417f4e","Type":"ContainerStarted","Data":"add3e5b512fbe1e77f25376da330ac4fbbe319bf6eeafa25471089143358027f"} Jul 30 03:46:35.075579 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.074841 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" event={"ID":"400d1899-9cd9-4c9a-96d8-1cff2e417f4e","Type":"ContainerStarted","Data":"c89cfc2b93e52f1231220ba1b1153023087387697506dd726b026c95c3b364dd"} Jul 30 03:46:35.075579 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.074869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" event={"ID":"400d1899-9cd9-4c9a-96d8-1cff2e417f4e","Type":"ContainerStarted","Data":"8d9360a54ec0b1cdbaf6511fe6477c2b81b8c84551ff91a21f8824e4dc76e4a1"} Jul 30 03:46:35.078223 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.078141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" event={"ID":"d75da48f-0d01-4a6f-96e5-fd01b7518337","Type":"ContainerStarted","Data":"4af4032bdc226a90eb5f8ac658e46b9052ad42edfa1574df6d0324041013a7b9"} Jul 30 03:46:35.081054 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.080971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" event={"ID":"1e90afbe-e730-42f7-9f25-7d203667789e","Type":"ContainerStarted","Data":"880c6279235dc93229bb3ab938dcec81d712166681f2c9cca6d798646f87b0dc"} Jul 30 03:46:35.082488 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.082433 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" event={"ID":"7db0b0f8-28b1-49c5-9772-ece2703fc684","Type":"ContainerStarted","Data":"198b63c9a637bfbb758c46f6386eb51f99e8c7ec997ae763b833c58344e994ad"} Jul 30 03:46:35.087176 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.086490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" event={"ID":"c300a760-a61e-4454-a5f2-98463d5387fe","Type":"ContainerStarted","Data":"a7343170a699110732b448a190bf7d54c1b02a212252c60f1b12d2bd86a52102"} Jul 30 03:46:35.091523 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.091495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"3134c7d2ffc15d556a097d4b6534522a021dfc5bbe3eb8fbfe3b69ed8dc44ccc"} Jul 30 03:46:35.091649 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.091571 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"77c79ac4e31e3ad02c0cd3f2aa8b213f3cadc6b665f8f570778b3c1f45118615"} Jul 30 03:46:35.091649 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.091610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"bdee163dc1310ac6b6a83892b71dd56e84d7f97888d6b2d84e525ba534e8ba87"} Jul 30 03:46:35.094273 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.093323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:35.094739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.094696 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6z88r" podStartSLOduration=4.651227854 podStartE2EDuration="7.094677848s" totalImagesPullingTime="2.443449994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:35.092672562 +0000 UTC m=+75.138171758" watchObservedRunningTime="2026-07-30 03:46:35.094677848 +0000 UTC m=+75.140177045" Jul 30 03:46:35.099381 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.099335 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px6sb" event={"ID":"230b0d8b-67b2-4e76-9fe9-858119ee618e","Type":"ContainerStarted","Data":"d2561eb248086a5f99bcc2dd9810bc762849fe12d95307c9abd73bd5b2752db0"} Jul 30 03:46:35.099381 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.099387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px6sb" event={"ID":"230b0d8b-67b2-4e76-9fe9-858119ee618e","Type":"ContainerStarted","Data":"1d833a93b1195f6a03f68f78e4fae50b3525d54ef60a4ab599e245cf36e5f396"} Jul 30 03:46:35.101956 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.101932 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef" exitCode=0 Jul 30 03:46:35.102066 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.101980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef"} Jul 30 03:46:35.110907 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.110847 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4sjgj" podStartSLOduration=4.808564047 podStartE2EDuration="7.110829889s" totalImagesPullingTime="2.302265842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:35.108853756 +0000 UTC m=+75.154352952" watchObservedRunningTime="2026-07-30 03:46:35.110829889 +0000 UTC m=+75.156329085" Jul 30 03:46:35.156229 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.155547 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-px6sb" podStartSLOduration=6.09506697 podStartE2EDuration="7.155524214s" totalImagesPullingTime="970.795182ms" totalInitContainerRuntime="89.662062ms" isStatefulPod=true podCreationTimestamp="2026-07-30 03:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:35.154051917 +0000 UTC m=+75.199551113" watchObservedRunningTime="2026-07-30 03:46:35.155524214 +0000 UTC m=+75.201023411" Jul 30 03:46:35.300690 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:35.300642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:46:35.518478 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:46:35.518419 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbaa0d1_df2d_47d6_afcf_6d2cb334dc00.slice/crio-c84a9f52d8bc9c851788ce93d6b9a187609ee0cc522c30ad3bae07ec47109481 WatchSource:0}: Error finding container c84a9f52d8bc9c851788ce93d6b9a187609ee0cc522c30ad3bae07ec47109481: Status 404 returned error can't find the container with id c84a9f52d8bc9c851788ce93d6b9a187609ee0cc522c30ad3bae07ec47109481 Jul 30 03:46:36.116446 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.115659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"d65504114f91606345d3e9ce672c953516287fe85bc71394c595540a945c852d"} Jul 30 03:46:36.116446 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.115755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"2471a93bba343b59fc588c99af080d802b2667dc653df9f379851f44a35e5735"} Jul 30 03:46:36.116446 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.115785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" event={"ID":"dd9ece92-d257-49c6-be38-4147864a1b66","Type":"ContainerStarted","Data":"80b8af6deed68654896d5221475c574e41a9e3745bc5bba9a768865296470a65"} Jul 30 03:46:36.116446 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.115871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:36.118497 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.117841 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" exitCode=0 Jul 30 03:46:36.119709 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.118757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} Jul 30 03:46:36.119709 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.118826 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"c84a9f52d8bc9c851788ce93d6b9a187609ee0cc522c30ad3bae07ec47109481"} Jul 30 03:46:36.142557 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:36.141928 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" podStartSLOduration=2.598929875 podStartE2EDuration="6.141909921s" totalImagesPullingTime="3.542980046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:36.138534221 +0000 UTC m=+76.184033416" watchObservedRunningTime="2026-07-30 03:46:36.141909921 +0000 UTC m=+76.187409116" Jul 30 03:46:38.142842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.142510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676"} Jul 30 03:46:38.142842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.142639 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3"} Jul 30 03:46:38.142842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.142683 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f"} Jul 30 03:46:38.142842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.142724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e"} Jul 30 03:46:38.142842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.142764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501"} Jul 30 03:46:38.147554 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.145110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" event={"ID":"d75da48f-0d01-4a6f-96e5-fd01b7518337","Type":"ContainerStarted","Data":"e47ba279a10e6ae32cea17647c8906099e2e15531c74ec68873551aa073a2861"} Jul 30 03:46:38.147554 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.146186 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:38.152331 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.152298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" event={"ID":"1e90afbe-e730-42f7-9f25-7d203667789e","Type":"ContainerStarted","Data":"8166175e3708756a5e6ab54a5ce86f228beb0e176c9ccf3d89811b19395904f9"} Jul 30 03:46:38.152435 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.152358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" event={"ID":"1e90afbe-e730-42f7-9f25-7d203667789e","Type":"ContainerStarted","Data":"de711ddd9f77c524e0785908cc2e4738c188c2333ed7d89b01bc34348afc11ef"} Jul 30 03:46:38.152435 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.152388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" event={"ID":"1e90afbe-e730-42f7-9f25-7d203667789e","Type":"ContainerStarted","Data":"201345a15f367290c86f928eb2c335ef49e17def6217b88a21e4693c567f5374"} Jul 30 03:46:38.153658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.153605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" Jul 30 03:46:38.155440 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.155412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" event={"ID":"7db0b0f8-28b1-49c5-9772-ece2703fc684","Type":"ContainerStarted","Data":"28a4514466916d74ad0f00960e78c90e0ed1527418f38ee4150811570ab473e8"} Jul 30 03:46:38.165794 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.165750 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-n5p2p" podStartSLOduration=1.844262694 podStartE2EDuration="5.165734253s" totalImagesPullingTime="3.321471559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:38.163971163 +0000 UTC m=+78.209470364" watchObservedRunningTime="2026-07-30 03:46:38.165734253 +0000 UTC m=+78.211233447" Jul 30 03:46:38.198237 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.198169 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9768d7845-5f4xm" podStartSLOduration=1.874033568 podStartE2EDuration="5.198130277s" totalImagesPullingTime="3.324096709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:38.195065144 +0000 UTC m=+78.240564336" watchObservedRunningTime="2026-07-30 03:46:38.198130277 +0000 UTC m=+78.243629473" Jul 30 03:46:38.214173 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:38.214114 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" podStartSLOduration=2.842749939 podStartE2EDuration="6.214090108s" totalImagesPullingTime="3.371340169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:38.212725898 +0000 UTC m=+78.258225089" watchObservedRunningTime="2026-07-30 03:46:38.214090108 +0000 UTC m=+78.259589304" Jul 30 03:46:39.167002 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:39.166957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerStarted","Data":"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0"} Jul 30 03:46:39.202271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:39.202209 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.160987653 podStartE2EDuration="10.202190984s" totalImagesPullingTime="5.002738462s" totalInitContainerRuntime="38.464869ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:39.197839517 +0000 UTC m=+79.243338755" watchObservedRunningTime="2026-07-30 03:46:39.202190984 +0000 UTC m=+79.247690234" Jul 30 03:46:41.180948 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.180911 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} Jul 30 03:46:41.181300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.180982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} Jul 30 03:46:41.181300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.181007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} Jul 30 03:46:41.181300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.181031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} Jul 30 03:46:41.181300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.181055 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} Jul 30 03:46:41.181300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.181077 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerStarted","Data":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} Jul 30 03:46:41.210517 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:41.210441 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.777220766 podStartE2EDuration="7.210417273s" totalImagesPullingTime="4.399155056s" totalInitContainerRuntime="34.041451ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:41.207421519 +0000 UTC m=+81.252920725" watchObservedRunningTime="2026-07-30 03:46:41.210417273 +0000 UTC m=+81.255916469" Jul 30 03:46:42.130990 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:42.130963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bd7d888b8-5xgx4" Jul 30 03:46:45.094455 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:45.094403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:46:53.166194 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:53.166158 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:46:53.166681 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:46:53.166220 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:47:03.061814 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:03.061781 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdg9f" Jul 30 03:47:07.714733 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:07.714707 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/init-config-reloader/0.log" Jul 30 03:47:07.914294 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:07.914262 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/alertmanager/0.log" Jul 30 03:47:08.114601 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:08.114527 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/config-reloader/0.log" Jul 30 03:47:08.313909 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:08.313884 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/kube-rbac-proxy-web/0.log" Jul 30 03:47:08.514279 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:08.514248 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/kube-rbac-proxy/0.log" Jul 30 03:47:08.714327 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:08.714301 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/kube-rbac-proxy-metric/0.log" Jul 30 03:47:08.913686 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:08.913615 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5893ca92-23fb-45a3-8079-3fe7f3768280/prom-label-proxy/0.log" Jul 30 03:47:09.314182 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:09.314153 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-state-metrics/0.log" Jul 30 03:47:09.514513 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:09.514489 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-rbac-proxy-main/0.log" Jul 30 03:47:09.713970 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:09.713940 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-rbac-proxy-self/0.log" Jul 30 03:47:09.913613 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:09.913548 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-684dd5868f-xm6zc_7db0b0f8-28b1-49c5-9772-ece2703fc684/metrics-server/0.log" Jul 30 03:47:10.113700 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:10.113621 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-n5p2p_d75da48f-0d01-4a6f-96e5-fd01b7518337/monitoring-plugin/0.log" Jul 30 03:47:10.913520 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:10.913493 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/init-textfile/0.log" Jul 30 03:47:11.114511 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:11.114454 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/node-exporter/0.log" Jul 30 03:47:11.313193 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:11.313163 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/kube-rbac-proxy/0.log" Jul 30 03:47:12.114233 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:12.114200 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/kube-rbac-proxy-main/0.log" Jul 30 03:47:12.313564 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:12.313535 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/kube-rbac-proxy-self/0.log" Jul 30 03:47:12.518156 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:12.518132 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/openshift-state-metrics/0.log" Jul 30 03:47:12.714309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:12.714282 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/init-config-reloader/0.log" Jul 30 03:47:12.915432 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:12.915409 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/prometheus/0.log" Jul 30 03:47:13.114629 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.114599 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/config-reloader/0.log" Jul 30 03:47:13.171921 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.171856 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:47:13.175624 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.175602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-684dd5868f-xm6zc" Jul 30 03:47:13.314449 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.314415 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/thanos-sidecar/0.log" Jul 30 03:47:13.514127 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.514047 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/kube-rbac-proxy-web/0.log" Jul 30 03:47:13.714085 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.714050 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/kube-rbac-proxy/0.log" Jul 30 03:47:13.914260 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:13.914187 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/kube-rbac-proxy-thanos/0.log" Jul 30 03:47:14.114640 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:14.114599 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v47mr_5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2/prometheus-operator/0.log" Jul 30 03:47:14.314060 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:14.314029 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v47mr_5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2/kube-rbac-proxy/0.log" Jul 30 03:47:14.513207 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:14.513167 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-7lf4s_f4cf6d70-21b0-4319-b424-07e4f8390b7b/prometheus-operator-admission-webhook/0.log" Jul 30 03:47:14.714105 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:14.714059 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/telemeter-client/0.log" Jul 30 03:47:14.914029 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:14.914000 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/reload/0.log" Jul 30 03:47:15.113961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:15.113875 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/kube-rbac-proxy/0.log" Jul 30 03:47:15.314738 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:15.314711 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/thanos-query/0.log" Jul 30 03:47:15.514157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:15.514127 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-web/0.log" Jul 30 03:47:15.713844 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:15.713818 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy/0.log" Jul 30 03:47:15.914147 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:15.914070 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/prom-label-proxy/0.log" Jul 30 03:47:16.113915 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:16.113890 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-rules/0.log" Jul 30 03:47:16.313732 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:16.313706 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-metrics/0.log" Jul 30 03:47:17.913256 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:17.913232 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d5kn_0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90/serve-healthcheck-canary/0.log" Jul 30 03:47:35.094652 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:35.094604 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:35.114785 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:35.114757 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:35.420599 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:35.420571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:48.869800 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.869767 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:48.870325 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870078 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="alertmanager" containerID="cri-o://870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" gracePeriod=120 Jul 30 03:47:48.870325 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870187 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-metric" containerID="cri-o://cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" gracePeriod=120 Jul 30 03:47:48.870325 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870226 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy" containerID="cri-o://f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" gracePeriod=120 Jul 30 03:47:48.870325 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870226 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="prom-label-proxy" containerID="cri-o://371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" gracePeriod=120 Jul 30 03:47:48.870325 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870234 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="config-reloader" containerID="cri-o://cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" gracePeriod=120 Jul 30 03:47:48.870654 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:48.870367 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-web" containerID="cri-o://bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" gracePeriod=120 Jul 30 03:47:49.460671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460639 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" exitCode=0 Jul 30 03:47:49.460671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460663 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" exitCode=0 Jul 30 03:47:49.460671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460670 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" exitCode=0 Jul 30 03:47:49.460671 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460676 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" exitCode=0 Jul 30 03:47:49.460929 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0"} Jul 30 03:47:49.460929 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460797 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3"} Jul 30 03:47:49.460929 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e"} Jul 30 03:47:49.460929 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:49.460848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501"} Jul 30 03:47:50.121044 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.121019 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.218854 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218768 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.218854 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.218854 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218845 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218870 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218900 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218915 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218936 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwj49\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219118 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.218955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219167 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 03:47:50.219440 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219383 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219531 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219436 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:50.219531 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219455 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219531 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219518 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.219847 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.219613 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5893ca92-23fb-45a3-8079-3fe7f3768280\" (UID: \"5893ca92-23fb-45a3-8079-3fe7f3768280\") " Jul 30 03:47:50.220067 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.220045 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-main-db\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.220137 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.220072 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.220973 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.220931 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:50.222589 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.222545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out" (OuterVolumeSpecName: "config-out") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 03:47:50.222967 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.222913 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49" (OuterVolumeSpecName: "kube-api-access-fwj49") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "kube-api-access-fwj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:47:50.223268 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.223228 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.223370 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.223344 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.223670 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.223640 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.223780 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.223710 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.223780 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.223758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:47:50.224588 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.224567 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume" (OuterVolumeSpecName: "config-volume") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.227213 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.227187 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.234906 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.234886 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config" (OuterVolumeSpecName: "web-config") pod "5893ca92-23fb-45a3-8079-3fe7f3768280" (UID: "5893ca92-23fb-45a3-8079-3fe7f3768280"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:50.320914 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320856 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5893ca92-23fb-45a3-8079-3fe7f3768280-metrics-client-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.320914 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320903 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.320914 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320915 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5893ca92-23fb-45a3-8079-3fe7f3768280-config-out\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.320914 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320924 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-main-tls\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.320914 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320934 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320942 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-web-config\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320950 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-cluster-tls-config\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320958 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj49\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-kube-api-access-fwj49\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320968 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5893ca92-23fb-45a3-8079-3fe7f3768280-tls-assets\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320976 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.321209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.320984 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5893ca92-23fb-45a3-8079-3fe7f3768280-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:50.466896 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.466861 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" exitCode=0 Jul 30 03:47:50.466896 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.466888 2568 generic.go:356] "Generic (PLEG): container finished" podID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerID="bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" exitCode=0 Jul 30 03:47:50.467095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.466944 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.467095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.466943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676"} Jul 30 03:47:50.467095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.467012 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f"} Jul 30 03:47:50.467095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.467041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5893ca92-23fb-45a3-8079-3fe7f3768280","Type":"ContainerDied","Data":"3d39aaf539c3f7a088b73d33c3a41d19867f84ea218da63b6fb02beb504753dd"} Jul 30 03:47:50.467095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.467083 2568 scope.go:117] "RemoveContainer" containerID="371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" Jul 30 03:47:50.475921 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.475899 2568 scope.go:117] "RemoveContainer" containerID="cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" Jul 30 03:47:50.483198 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.483179 2568 scope.go:117] "RemoveContainer" containerID="f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" Jul 30 03:47:50.489910 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.489889 2568 scope.go:117] "RemoveContainer" containerID="bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" Jul 30 03:47:50.493955 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.493928 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:50.498811 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.498791 2568 scope.go:117] "RemoveContainer" containerID="cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" Jul 30 03:47:50.498992 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.498970 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:50.506042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.506026 2568 scope.go:117] "RemoveContainer" containerID="870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" Jul 30 03:47:50.513582 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.513562 2568 scope.go:117] "RemoveContainer" containerID="560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef" Jul 30 03:47:50.520919 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.520901 2568 scope.go:117] "RemoveContainer" containerID="371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" Jul 30 03:47:50.521182 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.521164 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0\": container with ID starting with 371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0 not found: ID does not exist" containerID="371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" Jul 30 03:47:50.521229 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521195 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0"} err="failed to get container status \"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0\": rpc error: code = NotFound desc = could not find container \"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0\": container with ID starting with 371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0 not found: ID does not exist" Jul 30 03:47:50.521266 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521229 2568 scope.go:117] "RemoveContainer" containerID="cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" Jul 30 03:47:50.521489 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.521447 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676\": container with ID starting with cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676 not found: ID does not exist" containerID="cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" Jul 30 03:47:50.521555 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521498 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676"} err="failed to get container status \"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676\": rpc error: code = NotFound desc = could not find container \"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676\": container with ID starting with cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676 not found: ID does not exist" Jul 30 03:47:50.521555 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521516 2568 scope.go:117] "RemoveContainer" containerID="f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" Jul 30 03:47:50.521792 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.521774 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3\": container with ID starting with f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3 not found: ID does not exist" containerID="f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" Jul 30 03:47:50.521856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521798 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3"} err="failed to get container status \"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3\": rpc error: code = NotFound desc = could not find container \"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3\": container with ID starting with f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3 not found: ID does not exist" Jul 30 03:47:50.521856 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.521820 2568 scope.go:117] "RemoveContainer" containerID="bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" Jul 30 03:47:50.522039 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.522023 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f\": container with ID starting with bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f not found: ID does not exist" containerID="bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" Jul 30 03:47:50.522098 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522042 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f"} err="failed to get container status \"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f\": rpc error: code = NotFound desc = could not find container \"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f\": container with ID starting with bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f not found: ID does not exist" Jul 30 03:47:50.522098 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522054 2568 scope.go:117] "RemoveContainer" containerID="cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" Jul 30 03:47:50.522289 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.522268 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e\": container with ID starting with cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e not found: ID does not exist" containerID="cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" Jul 30 03:47:50.522350 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522295 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e"} err="failed to get container status \"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e\": rpc error: code = NotFound desc = could not find container \"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e\": container with ID starting with cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e not found: ID does not exist" Jul 30 03:47:50.522350 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522314 2568 scope.go:117] "RemoveContainer" containerID="870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" Jul 30 03:47:50.522642 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.522588 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501\": container with ID starting with 870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501 not found: ID does not exist" containerID="870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" Jul 30 03:47:50.522733 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522649 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501"} err="failed to get container status \"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501\": rpc error: code = NotFound desc = could not find container \"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501\": container with ID starting with 870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501 not found: ID does not exist" Jul 30 03:47:50.522733 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.522665 2568 scope.go:117] "RemoveContainer" containerID="560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef" Jul 30 03:47:50.523122 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:50.523076 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef\": container with ID starting with 560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef not found: ID does not exist" containerID="560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef" Jul 30 03:47:50.523230 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523207 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef"} err="failed to get container status \"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef\": rpc error: code = NotFound desc = could not find container \"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef\": container with ID starting with 560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef not found: ID does not exist" Jul 30 03:47:50.523312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523235 2568 scope.go:117] "RemoveContainer" containerID="371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0" Jul 30 03:47:50.523726 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523684 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0"} err="failed to get container status \"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0\": rpc error: code = NotFound desc = could not find container \"371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0\": container with ID starting with 371e4f620dda2a4cccc43a7c17051d914f07cd11d32a72ca5cd1a4b3c69078a0 not found: ID does not exist" Jul 30 03:47:50.523726 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523717 2568 scope.go:117] "RemoveContainer" containerID="cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676" Jul 30 03:47:50.523989 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523964 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676"} err="failed to get container status \"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676\": rpc error: code = NotFound desc = could not find container \"cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676\": container with ID starting with cf36b2d691a5407881dfd49ac05c42f793bcfeab3d7e425b14416507ecb06676 not found: ID does not exist" Jul 30 03:47:50.524059 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.523991 2568 scope.go:117] "RemoveContainer" containerID="f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3" Jul 30 03:47:50.524226 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524209 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3"} err="failed to get container status \"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3\": rpc error: code = NotFound desc = could not find container \"f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3\": container with ID starting with f85e35a7acff12cc53f68db6e8bf3a22efd59a6fea7fd0c058950aa84dab57b3 not found: ID does not exist" Jul 30 03:47:50.524291 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524227 2568 scope.go:117] "RemoveContainer" containerID="bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f" Jul 30 03:47:50.524447 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524427 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f"} err="failed to get container status \"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f\": rpc error: code = NotFound desc = could not find container \"bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f\": container with ID starting with bac51dccdf11ec32c062e2a263883e31959e4af92ffee4f42624a62d0b4c974f not found: ID does not exist" Jul 30 03:47:50.524525 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524450 2568 scope.go:117] "RemoveContainer" containerID="cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e" Jul 30 03:47:50.524738 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524711 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e"} err="failed to get container status \"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e\": rpc error: code = NotFound desc = could not find container \"cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e\": container with ID starting with cd71760b7616e8c5d064d27784a92900bc673c48310929b80e6c3750d410e04e not found: ID does not exist" Jul 30 03:47:50.524813 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524740 2568 scope.go:117] "RemoveContainer" containerID="870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501" Jul 30 03:47:50.524979 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524963 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501"} err="failed to get container status \"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501\": rpc error: code = NotFound desc = could not find container \"870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501\": container with ID starting with 870fa683c92639e4c62c5b28e53bbdffdedac206b637852c796b20fe6024b501 not found: ID does not exist" Jul 30 03:47:50.525026 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.524980 2568 scope.go:117] "RemoveContainer" containerID="560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef" Jul 30 03:47:50.525026 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525003 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:50.525174 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525151 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef"} err="failed to get container status \"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef\": rpc error: code = NotFound desc = could not find container \"560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef\": container with ID starting with 560474fd3b1f835498e08c3dc824e1a6384e8644bed384f2b50a16bcc2f357ef not found: ID does not exist" Jul 30 03:47:50.525337 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525324 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="config-reloader" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525340 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="config-reloader" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525348 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="prom-label-proxy" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525354 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="prom-label-proxy" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525362 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-web" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525367 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-web" Jul 30 03:47:50.525375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525375 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="alertmanager" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525382 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="alertmanager" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525389 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-metric" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525394 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-metric" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525401 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="init-config-reloader" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525406 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="init-config-reloader" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525416 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525421 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525490 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-web" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525500 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy-metric" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525509 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="kube-rbac-proxy" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525519 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="alertmanager" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525525 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="prom-label-proxy" Jul 30 03:47:50.525615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.525536 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" containerName="config-reloader" Jul 30 03:47:50.529447 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.529426 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.531915 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.531897 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 03:47:50.532032 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532006 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 03:47:50.532095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532042 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 03:47:50.532095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532065 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 03:47:50.532095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 03:47:50.532256 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 03:47:50.532308 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532298 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 03:47:50.532503 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532490 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 03:47:50.532550 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.532518 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qn7td\"" Jul 30 03:47:50.537615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.537594 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 03:47:50.542285 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.542263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:50.568204 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.568165 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5893ca92-23fb-45a3-8079-3fe7f3768280" path="/var/lib/kubelet/pods/5893ca92-23fb-45a3-8079-3fe7f3768280/volumes" Jul 30 03:47:50.623733 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2g7\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-kube-api-access-xt2g7\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.623903 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.623974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.624001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-out\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.624026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-web-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.624129 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.624068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.724857 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.724747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.724857 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.724841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-out\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.724889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-web-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.724978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725167 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2g7\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-kube-api-access-xt2g7\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.725395 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.725344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.726720 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.726692 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728277 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728393 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728393 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728256 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-out\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728393 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728380 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728590 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728380 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-web-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.728712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.728658 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.729329 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.729140 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.729569 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.729539 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ffbea0b-ece0-4af3-bd33-5543d36007c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.730800 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.730772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.731001 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.730976 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.731073 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.731010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ffbea0b-ece0-4af3-bd33-5543d36007c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.733014 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.732993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2g7\" (UniqueName: \"kubernetes.io/projected/2ffbea0b-ece0-4af3-bd33-5543d36007c4-kube-api-access-xt2g7\") pod \"alertmanager-main-0\" (UID: \"2ffbea0b-ece0-4af3-bd33-5543d36007c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.838998 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.838965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 03:47:50.979024 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:50.978996 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 03:47:50.981656 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:47:50.981623 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffbea0b_ece0_4af3_bd33_5543d36007c4.slice/crio-db6672e4fb7d0c69fcb20222a4def2b72077463b5c57f2f81ad7630b53062b8b WatchSource:0}: Error finding container db6672e4fb7d0c69fcb20222a4def2b72077463b5c57f2f81ad7630b53062b8b: Status 404 returned error can't find the container with id db6672e4fb7d0c69fcb20222a4def2b72077463b5c57f2f81ad7630b53062b8b Jul 30 03:47:51.472617 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:51.472583 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ffbea0b-ece0-4af3-bd33-5543d36007c4" containerID="a2fe2d99c12b76e45f58ba1d9ed1cbc64c38f496b320a983abff271dc1f12847" exitCode=0 Jul 30 03:47:51.473070 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:51.472676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerDied","Data":"a2fe2d99c12b76e45f58ba1d9ed1cbc64c38f496b320a983abff271dc1f12847"} Jul 30 03:47:51.473070 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:51.472755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"db6672e4fb7d0c69fcb20222a4def2b72077463b5c57f2f81ad7630b53062b8b"} Jul 30 03:47:52.481518 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481479 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"c10e679f521df59f222854fb2e2813c0edf3e6001f19828b2bd1b4d79d7d288f"} Jul 30 03:47:52.482008 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"e2d162367f7276212bb9a6d61bb14bac8136155e60367725e994ac9321459a97"} Jul 30 03:47:52.482008 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"99f6a7c57fcfc66a0f543b41a2252bc2927332662b41b2f98d474fd665e81d80"} Jul 30 03:47:52.482008 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"aa49d8ca9c56c8103bc855a2c377d2ab42680b4929f9eca8a5d02f76c9036d23"} Jul 30 03:47:52.482008 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"6aac8acf10da7fa30da21a66a1e97b07a400a54981e96dcad3cbf36ee00b841c"} Jul 30 03:47:52.482008 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.481722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ffbea0b-ece0-4af3-bd33-5543d36007c4","Type":"ContainerStarted","Data":"af8bf1dc6c9cc88ead05db17ff549b831f3b5bff6a505a18ff4bedd9ca2ffa8a"} Jul 30 03:47:52.508487 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:52.508412 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.490441347 podStartE2EDuration="2.508392899s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.951552ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:47:52.507143706 +0000 UTC m=+152.552642901" watchObservedRunningTime="2026-07-30 03:47:52.508392899 +0000 UTC m=+152.553892094" Jul 30 03:47:53.114062 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114024 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:53.114705 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114658 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="prometheus" containerID="cri-o://d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" gracePeriod=600 Jul 30 03:47:53.114853 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114684 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="thanos-sidecar" containerID="cri-o://a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" gracePeriod=600 Jul 30 03:47:53.114853 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114726 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-thanos" containerID="cri-o://64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" gracePeriod=600 Jul 30 03:47:53.114853 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114809 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="config-reloader" containerID="cri-o://287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" gracePeriod=600 Jul 30 03:47:53.115021 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114865 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy" containerID="cri-o://9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" gracePeriod=600 Jul 30 03:47:53.115021 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.114889 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-web" containerID="cri-o://dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" gracePeriod=600 Jul 30 03:47:53.367794 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.367720 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.449794 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449753 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.449980 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.449980 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449846 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z42xj\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.449980 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449906 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.449980 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449937 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.449980 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449960 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.449984 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450017 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450045 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450085 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450118 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450149 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450178 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450234 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450223 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450255 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450310 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450334 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450370 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450412 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.450663 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.450445 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca\") pod \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\" (UID: \"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00\") " Jul 30 03:47:53.451037 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.451008 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.451608 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.451527 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.452010 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.451950 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.452862 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.452620 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.452862 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.452725 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.452862 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.452816 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.453053 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.452896 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.453053 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.453031 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj" (OuterVolumeSpecName: "kube-api-access-z42xj") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "kube-api-access-z42xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:47:53.453053 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.453035 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.454227 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.453715 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 03:47:53.454227 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.453880 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out" (OuterVolumeSpecName: "config-out") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 03:47:53.454227 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.454183 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.454414 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.454387 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 03:47:53.455038 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.455013 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.455380 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.455354 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.456016 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.455996 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.456108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.456059 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.456375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.456350 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:47:53.456375 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.456351 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config" (OuterVolumeSpecName: "config") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.465015 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.464988 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config" (OuterVolumeSpecName: "web-config") pod "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" (UID: "7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:47:53.488741 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488709 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" exitCode=0 Jul 30 03:47:53.488741 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488734 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" exitCode=0 Jul 30 03:47:53.488741 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488741 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" exitCode=0 Jul 30 03:47:53.488741 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488747 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" exitCode=0 Jul 30 03:47:53.488741 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488752 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" exitCode=0 Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488757 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" exitCode=0 Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488783 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.489011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.488959 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.489238 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.489046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00","Type":"ContainerDied","Data":"c84a9f52d8bc9c851788ce93d6b9a187609ee0cc522c30ad3bae07ec47109481"} Jul 30 03:47:53.497656 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.497635 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.504568 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.504548 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.512701 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.512649 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:53.512933 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.512915 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.517065 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.517039 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:53.520492 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.520453 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.527312 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.527295 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.534675 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.534651 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.541664 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.541640 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:53.541964 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.541947 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.542120 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542087 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="init-config-reloader" Jul 30 03:47:53.542173 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542120 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="init-config-reloader" Jul 30 03:47:53.542173 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542136 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-web" Jul 30 03:47:53.542173 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542145 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-web" Jul 30 03:47:53.542173 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542168 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542177 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542186 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-thanos" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542194 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-thanos" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542207 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="thanos-sidecar" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542214 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="thanos-sidecar" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542237 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="config-reloader" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.542234 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542266 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.542309 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542286 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542243 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="config-reloader" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542333 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="prometheus" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542341 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="prometheus" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542414 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="thanos-sidecar" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542426 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-web" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542438 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="prometheus" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542445 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542454 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="kube-rbac-proxy-thanos" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542483 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" containerName="config-reloader" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.542537 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542560 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.542658 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542573 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.542816 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542849 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.542866 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.543092 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543109 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.543199 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543121 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.543424 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.543323 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.543424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543339 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.543424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543351 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.543842 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.543823 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.543884 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543846 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.543884 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.543861 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.544070 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:47:53.544055 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.544104 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544073 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.544104 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544084 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.544316 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544298 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.544364 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544318 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.544537 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544518 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.544602 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544540 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.544769 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544749 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.544834 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544770 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.544958 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544941 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.545006 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.544959 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.545165 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545147 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.545165 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545164 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.545371 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545352 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.545447 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545374 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.545672 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545647 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.545797 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545673 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.546124 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.545987 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.546124 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546015 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.546327 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546305 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.546387 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546329 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.546626 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546604 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.546626 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546627 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.546906 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546886 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.546989 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546906 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.546989 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.546912 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.547191 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547165 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.547191 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547190 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.547505 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547457 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.547587 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547507 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.547783 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547763 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.547783 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.547782 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.548062 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548036 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.548138 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548062 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.548352 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548328 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.548412 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548357 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.548678 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548658 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.548678 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548678 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.548929 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548911 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.548973 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.548930 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.549174 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549157 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.549242 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549185 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549643 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549658 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549668 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549749 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549758 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549658 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aln0ul09ge2m3\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549869 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 03:47:53.549943 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549871 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 03:47:53.550483 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.549993 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 03:47:53.550483 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550041 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 03:47:53.550483 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550045 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 03:47:53.550483 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550361 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.550483 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550378 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.550810 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 03:47:53.550862 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550847 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.550921 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550866 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.550990 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.550973 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lbnhs\"" Jul 30 03:47:53.554141 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554118 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42xj\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-kube-api-access-z42xj\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554144 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-web-config\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554157 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config-out\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554170 2568 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-kube-rbac-proxy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554183 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554197 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554210 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554223 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554239 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554261 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554275 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-tls-assets\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554290 2568 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554302 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-db\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554317 2568 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-config\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554330 2568 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-metrics-client-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554343 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554357 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554370 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-configmap-metrics-client-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554382 2568 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-grpc-tls\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.554739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.554395 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:47:53.555598 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.555571 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.555598 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.555590 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 03:47:53.555598 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.555600 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.556016 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.555914 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.556016 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.555940 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.556708 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.556676 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.556800 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.556706 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.556982 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.556963 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.556982 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.556982 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.557171 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557150 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 03:47:53.557233 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557209 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.557233 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557230 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.557530 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557505 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.557622 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557530 2568 scope.go:117] "RemoveContainer" containerID="64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7" Jul 30 03:47:53.557755 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557737 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7"} err="failed to get container status \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": rpc error: code = NotFound desc = could not find container \"64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7\": container with ID starting with 64baf83a3b4a7d44bffc6f3a40963014007a477ac0dda6edfb1742181a6b31b7 not found: ID does not exist" Jul 30 03:47:53.557755 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557754 2568 scope.go:117] "RemoveContainer" containerID="9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06" Jul 30 03:47:53.557960 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557942 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06"} err="failed to get container status \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": rpc error: code = NotFound desc = could not find container \"9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06\": container with ID starting with 9d9d4e9766563e1ed7123b7c81039d48e65256816998507b6604c0a17fdfde06 not found: ID does not exist" Jul 30 03:47:53.557960 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.557958 2568 scope.go:117] "RemoveContainer" containerID="dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2" Jul 30 03:47:53.558172 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558155 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2"} err="failed to get container status \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": rpc error: code = NotFound desc = could not find container \"dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2\": container with ID starting with dab7c1eb5f7138f3e6e415d19674b2623462b197b6f9a1dc0f8149339bc6e4e2 not found: ID does not exist" Jul 30 03:47:53.558229 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558179 2568 scope.go:117] "RemoveContainer" containerID="a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1" Jul 30 03:47:53.558403 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558388 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1"} err="failed to get container status \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": rpc error: code = NotFound desc = could not find container \"a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1\": container with ID starting with a92d28088acac4ecfbe86d3285229a9fa61ddfd47ecdb3b09dd0046e2c31b2e1 not found: ID does not exist" Jul 30 03:47:53.558452 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558403 2568 scope.go:117] "RemoveContainer" containerID="287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a" Jul 30 03:47:53.558643 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558623 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a"} err="failed to get container status \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": rpc error: code = NotFound desc = could not find container \"287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a\": container with ID starting with 287c5766450db0e24c49add501e9b3d4bd962ba2c28786b44d5370279a84e46a not found: ID does not exist" Jul 30 03:47:53.558696 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558644 2568 scope.go:117] "RemoveContainer" containerID="d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d" Jul 30 03:47:53.558868 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558852 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d"} err="failed to get container status \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": rpc error: code = NotFound desc = could not find container \"d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d\": container with ID starting with d0418f1b1c9298279485ca37dd99d677fa4e01b62f30202da64edad966e3f71d not found: ID does not exist" Jul 30 03:47:53.558912 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.558868 2568 scope.go:117] "RemoveContainer" containerID="ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062" Jul 30 03:47:53.559121 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.559097 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062"} err="failed to get container status \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": rpc error: code = NotFound desc = could not find container \"ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062\": container with ID starting with ec1afa4b49bb6edf88efc950e0f58b917b646cbdff4bbe2cc2d6944b1e79b062 not found: ID does not exist" Jul 30 03:47:53.561725 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.561701 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:53.655427 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655427 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655427 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655427 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655604 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655691 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6lt\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-kube-api-access-cj6lt\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655805 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.655920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.656131 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.655919 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.756873 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.756873 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.756994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757055 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757119 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6lt\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-kube-api-access-cj6lt\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757357 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757399 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.757646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.758140 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.758140 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.758140 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.757829 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.759888 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.758920 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.759888 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.759562 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.760482 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.760429 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.760592 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.760509 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.760655 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.760619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.760711 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.760653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe0c8641-b074-4a2e-8901-5bcc3da10403-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.760767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.760714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.761065 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.761040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.761549 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.761520 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.762085 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.762057 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe0c8641-b074-4a2e-8901-5bcc3da10403-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.762202 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.762183 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-config\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.763014 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.762993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.763369 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.763345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.763452 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.763376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe0c8641-b074-4a2e-8901-5bcc3da10403-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.763648 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.763632 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.764605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.764586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6lt\" (UniqueName: \"kubernetes.io/projected/fe0c8641-b074-4a2e-8901-5bcc3da10403-kube-api-access-cj6lt\") pod \"prometheus-k8s-0\" (UID: \"fe0c8641-b074-4a2e-8901-5bcc3da10403\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:53.859426 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.859385 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:47:54.001066 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:53.998531 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 03:47:54.004286 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:47:54.004236 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0c8641_b074_4a2e_8901_5bcc3da10403.slice/crio-88816031b5811b29f519f6d3d93b569ff79377a82a25fa52aa13900ad0c42275 WatchSource:0}: Error finding container 88816031b5811b29f519f6d3d93b569ff79377a82a25fa52aa13900ad0c42275: Status 404 returned error can't find the container with id 88816031b5811b29f519f6d3d93b569ff79377a82a25fa52aa13900ad0c42275 Jul 30 03:47:54.495598 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:54.495563 2568 generic.go:356] "Generic (PLEG): container finished" podID="fe0c8641-b074-4a2e-8901-5bcc3da10403" containerID="42d31a4a277ecf4c44479f26e0cbe6dd04b169d764804169ae66f1754059d5e3" exitCode=0 Jul 30 03:47:54.496055 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:54.495626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerDied","Data":"42d31a4a277ecf4c44479f26e0cbe6dd04b169d764804169ae66f1754059d5e3"} Jul 30 03:47:54.496055 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:54.495675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"88816031b5811b29f519f6d3d93b569ff79377a82a25fa52aa13900ad0c42275"} Jul 30 03:47:54.569769 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:54.569738 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00" path="/var/lib/kubelet/pods/7fbaa0d1-df2d-47d6-afcf-6d2cb334dc00/volumes" Jul 30 03:47:55.503206 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503174 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"7336a2168adb14e3b2eee45ac34f1ccf0bdfcc113185d2aa60b3af36eb75795b"} Jul 30 03:47:55.503605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"ef9d6807718c975aae4173cf369f652b824161b08816af13df1044ea93862b76"} Jul 30 03:47:55.503605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"eeebd4a9a8108dd1ca94e61b8159fd875af95c7ae0927a37d7fc2d2c4ed50871"} Jul 30 03:47:55.503605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"4469d8e1cfa9fd279c6b3fb91346193ae858b636cf701064603810e5e0c18ad9"} Jul 30 03:47:55.503605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503317 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"a9ae695830e4109ee51ed4876da046ff9a9c7a81e44ac4fb1d0c2f24d8b9ec70"} Jul 30 03:47:55.503605 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.503338 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe0c8641-b074-4a2e-8901-5bcc3da10403","Type":"ContainerStarted","Data":"4d695848e41d77e6cf155109b2a93205bb716a4dca518901120fa64ad521f839"} Jul 30 03:47:55.529955 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:55.529894 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5104206749999998 podStartE2EDuration="2.529872353s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.451678ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:47:55.528910065 +0000 UTC m=+155.574409260" watchObservedRunningTime="2026-07-30 03:47:55.529872353 +0000 UTC m=+155.575371548" Jul 30 03:47:58.859842 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:47:58.859791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:48:52.049342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.049303 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsftt"] Jul 30 03:48:52.052271 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.052247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.054292 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.054268 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j7lqj\"" Jul 30 03:48:52.054740 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.054723 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 03:48:52.054928 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.054913 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 03:48:52.062772 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.062749 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsftt"] Jul 30 03:48:52.206191 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.206147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wt95\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-kube-api-access-6wt95\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.206191 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.206196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.307392 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.307280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wt95\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-kube-api-access-6wt95\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.307392 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.307349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.314990 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.314963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.315131 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.315103 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wt95\" (UniqueName: \"kubernetes.io/projected/c1fc1101-2821-4a13-8d24-726d43374400-kube-api-access-6wt95\") pod \"cert-manager-webhook-6486ffd997-wsftt\" (UID: \"c1fc1101-2821-4a13-8d24-726d43374400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.373454 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.373418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:52.502428 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.502402 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsftt"] Jul 30 03:48:52.505166 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:48:52.505139 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1fc1101_2821_4a13_8d24_726d43374400.slice/crio-9ed870832af08941d89c34f5257ee5bbffc0292bfec8e89a39856613a842692e WatchSource:0}: Error finding container 9ed870832af08941d89c34f5257ee5bbffc0292bfec8e89a39856613a842692e: Status 404 returned error can't find the container with id 9ed870832af08941d89c34f5257ee5bbffc0292bfec8e89a39856613a842692e Jul 30 03:48:52.734752 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:52.734711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" event={"ID":"c1fc1101-2821-4a13-8d24-726d43374400","Type":"ContainerStarted","Data":"9ed870832af08941d89c34f5257ee5bbffc0292bfec8e89a39856613a842692e"} Jul 30 03:48:53.860125 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:53.860066 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:48:53.882661 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:53.882610 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:48:54.765855 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:54.765818 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 03:48:56.756212 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:56.756176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" event={"ID":"c1fc1101-2821-4a13-8d24-726d43374400","Type":"ContainerStarted","Data":"542343d58ba5a7f8a02d3f2386f72af99c63d4d7af0c3a830aa13263baef8a78"} Jul 30 03:48:56.756620 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:56.756313 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:48:56.771401 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:56.771354 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" podStartSLOduration=1.44073202 podStartE2EDuration="4.771339627s" totalImagesPullingTime="3.330607607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:48:56.770139569 +0000 UTC m=+216.815638764" watchObservedRunningTime="2026-07-30 03:48:56.771339627 +0000 UTC m=+216.816838821" Jul 30 03:48:57.254405 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.254344 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8"] Jul 30 03:48:57.256654 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.256636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.259099 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.259079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-768s4\"" Jul 30 03:48:57.266479 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.266445 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8"] Jul 30 03:48:57.359439 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.359397 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.359739 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.359453 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjh78\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-kube-api-access-wjh78\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.460275 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.460236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.460449 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.460288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjh78\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-kube-api-access-wjh78\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.468100 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.468062 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.468222 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.468129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjh78\" (UniqueName: \"kubernetes.io/projected/17236a8a-24e4-4b27-9c51-d61e72431a87-kube-api-access-wjh78\") pod \"cert-manager-cainjector-559cdd8c87-jfrx8\" (UID: \"17236a8a-24e4-4b27-9c51-d61e72431a87\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.565646 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.565549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" Jul 30 03:48:57.696292 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.696267 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8"] Jul 30 03:48:57.698607 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:48:57.698579 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17236a8a_24e4_4b27_9c51_d61e72431a87.slice/crio-f1195f218520356c23882ea31008b9734fe683bf20d112a49c7c04bb575f18d7 WatchSource:0}: Error finding container f1195f218520356c23882ea31008b9734fe683bf20d112a49c7c04bb575f18d7: Status 404 returned error can't find the container with id f1195f218520356c23882ea31008b9734fe683bf20d112a49c7c04bb575f18d7 Jul 30 03:48:57.766280 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.766185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" event={"ID":"17236a8a-24e4-4b27-9c51-d61e72431a87","Type":"ContainerStarted","Data":"03782f56901b1e49df0d6e3c411ba0311d3773fbbcce945a745f79525a3bc114"} Jul 30 03:48:57.766280 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.766243 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" event={"ID":"17236a8a-24e4-4b27-9c51-d61e72431a87","Type":"ContainerStarted","Data":"f1195f218520356c23882ea31008b9734fe683bf20d112a49c7c04bb575f18d7"} Jul 30 03:48:57.781011 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:48:57.780964 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jfrx8" podStartSLOduration=0.780949203 podStartE2EDuration="780.949203ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:48:57.780115446 +0000 UTC m=+217.825614642" watchObservedRunningTime="2026-07-30 03:48:57.780949203 +0000 UTC m=+217.826448398" Jul 30 03:49:02.768779 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:02.768748 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wsftt" Jul 30 03:49:18.729294 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.729256 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv"] Jul 30 03:49:18.732423 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.732403 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.736128 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736103 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 30 03:49:18.736263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736106 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 30 03:49:18.736263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736163 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 30 03:49:18.736263 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736185 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mhvrh\"" Jul 30 03:49:18.736440 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736370 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 03:49:18.736440 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.736393 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 03:49:18.742737 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.742714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv"] Jul 30 03:49:18.851249 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.851214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-manager-config\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.851448 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.851293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtsv\" (UniqueName: \"kubernetes.io/projected/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-kube-api-access-tmtsv\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.851448 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.851328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-metrics-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.851448 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.851348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.952300 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.952265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtsv\" (UniqueName: \"kubernetes.io/projected/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-kube-api-access-tmtsv\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.952508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.952305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-metrics-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.952508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.952333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.952508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.952359 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-manager-config\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.953145 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.953125 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-manager-config\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.954872 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.954847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-metrics-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.954962 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.954887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-cert\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:18.960209 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:18.960190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtsv\" (UniqueName: \"kubernetes.io/projected/b7e259d5-da3e-4b53-b2b9-8f6ac377fab4-kube-api-access-tmtsv\") pod \"lws-controller-manager-69d4f5b96d-ghlvv\" (UID: \"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4\") " pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:19.043734 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.043630 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:19.183544 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.183378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv"] Jul 30 03:49:19.186322 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:49:19.186295 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e259d5_da3e_4b53_b2b9_8f6ac377fab4.slice/crio-a2708f65caddde9dd9b2e4a3ebcb55716b09e5f9833b272a6aa150a75ba102f5 WatchSource:0}: Error finding container a2708f65caddde9dd9b2e4a3ebcb55716b09e5f9833b272a6aa150a75ba102f5: Status 404 returned error can't find the container with id a2708f65caddde9dd9b2e4a3ebcb55716b09e5f9833b272a6aa150a75ba102f5 Jul 30 03:49:19.844829 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.844798 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq"] Jul 30 03:49:19.849092 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.849071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:19.852756 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.852721 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mj4sc\"" Jul 30 03:49:19.852906 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.852804 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 03:49:19.852906 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.852832 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 03:49:19.852906 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.852902 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 03:49:19.853081 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.853001 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 03:49:19.854804 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.854780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" event={"ID":"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4","Type":"ContainerStarted","Data":"a2708f65caddde9dd9b2e4a3ebcb55716b09e5f9833b272a6aa150a75ba102f5"} Jul 30 03:49:19.866040 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.866019 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq"] Jul 30 03:49:19.961725 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.961684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:19.961897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.961733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:19.961897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.961787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:19.961897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.961823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:19.961897 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:19.961875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x58k\" (UniqueName: \"kubernetes.io/projected/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-kube-api-access-8x58k\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.062712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.062671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.062712 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.062710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x58k\" (UniqueName: \"kubernetes.io/projected/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-kube-api-access-8x58k\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.062923 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.062774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.062923 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.062799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.062923 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.062831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.063305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.063273 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.065331 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.065300 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.065447 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.065362 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.065447 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.065428 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.083968 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.083936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x58k\" (UniqueName: \"kubernetes.io/projected/8d49e557-b78d-4e43-8cc8-b1ff47c865f2-kube-api-access-8x58k\") pod \"opendatahub-operator-controller-manager-599789d45f-tp7gq\" (UID: \"8d49e557-b78d-4e43-8cc8-b1ff47c865f2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.161492 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.161384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:20.316099 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.315932 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq"] Jul 30 03:49:20.318599 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:49:20.318572 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d49e557_b78d_4e43_8cc8_b1ff47c865f2.slice/crio-b0fdb065e9a3ee99a4882236e4a126773db795e74c2b7d06d263132128f6e5fb WatchSource:0}: Error finding container b0fdb065e9a3ee99a4882236e4a126773db795e74c2b7d06d263132128f6e5fb: Status 404 returned error can't find the container with id b0fdb065e9a3ee99a4882236e4a126773db795e74c2b7d06d263132128f6e5fb Jul 30 03:49:20.862503 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:20.862443 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" event={"ID":"8d49e557-b78d-4e43-8cc8-b1ff47c865f2","Type":"ContainerStarted","Data":"b0fdb065e9a3ee99a4882236e4a126773db795e74c2b7d06d263132128f6e5fb"} Jul 30 03:49:24.885083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:24.885038 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" event={"ID":"b7e259d5-da3e-4b53-b2b9-8f6ac377fab4","Type":"ContainerStarted","Data":"6a7cf4d9077a9edc66ad3474521b10482e5c9b0e60cc05ff5646fa6fc80abe3e"} Jul 30 03:49:24.885615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:24.885157 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:24.886719 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:24.886696 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" event={"ID":"8d49e557-b78d-4e43-8cc8-b1ff47c865f2","Type":"ContainerStarted","Data":"3da7b1c9c6345a92441e24e86cf1dde3f79366a05af9b2af8e7900c1a72fbe5b"} Jul 30 03:49:24.899754 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:24.899699 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" podStartSLOduration=2.222822172 podStartE2EDuration="6.899682131s" totalImagesPullingTime="4.676859959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:24.899338313 +0000 UTC m=+244.944837511" watchObservedRunningTime="2026-07-30 03:49:24.899682131 +0000 UTC m=+244.945181327" Jul 30 03:49:25.892926 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:25.892891 2568 generic.go:356] "Generic (PLEG): container finished" podID="8d49e557-b78d-4e43-8cc8-b1ff47c865f2" containerID="3da7b1c9c6345a92441e24e86cf1dde3f79366a05af9b2af8e7900c1a72fbe5b" exitCode=0 Jul 30 03:49:25.893338 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:25.892977 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" event={"ID":"8d49e557-b78d-4e43-8cc8-b1ff47c865f2","Type":"ContainerDied","Data":"3da7b1c9c6345a92441e24e86cf1dde3f79366a05af9b2af8e7900c1a72fbe5b"} Jul 30 03:49:26.899107 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:26.899066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" event={"ID":"8d49e557-b78d-4e43-8cc8-b1ff47c865f2","Type":"ContainerStarted","Data":"ac094db9c1bb185ad34cea137933ee6a6fee8f2234b083ae6eeee37d4d77054f"} Jul 30 03:49:26.899516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:26.899290 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:26.918105 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:26.918025 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" podStartSLOduration=2.829517674 podStartE2EDuration="7.918005383s" totalImagesPullingTime="4.028463825s" totalInitContainerRuntime="1.060023884s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:26.916207443 +0000 UTC m=+246.961706639" watchObservedRunningTime="2026-07-30 03:49:26.918005383 +0000 UTC m=+246.963504578" Jul 30 03:49:35.895198 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:35.895164 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69d4f5b96d-ghlvv" Jul 30 03:49:37.905826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:37.905799 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-tp7gq" Jul 30 03:49:47.269717 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.269635 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6"] Jul 30 03:49:47.273301 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.273277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.276114 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.276082 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 03:49:47.276246 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.276163 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 03:49:47.276357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.276337 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r9r2l\"" Jul 30 03:49:47.276540 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.276523 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 03:49:47.291935 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.291907 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6"] Jul 30 03:49:47.318101 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318101 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779c70c3-62de-4d87-b035-1ac37190fb17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318307 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxl7\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-kube-api-access-xnxl7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318556 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318361 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.318556 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.318415 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779c70c3-62de-4d87-b035-1ac37190fb17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420004 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.419967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779c70c3-62de-4d87-b035-1ac37190fb17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779c70c3-62de-4d87-b035-1ac37190fb17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxl7\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-kube-api-access-xnxl7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420508 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420770 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420770 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420883 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420805 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779c70c3-62de-4d87-b035-1ac37190fb17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.420883 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420865 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.421019 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.420998 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.422920 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.422894 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779c70c3-62de-4d87-b035-1ac37190fb17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.423011 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.422927 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779c70c3-62de-4d87-b035-1ac37190fb17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.431168 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.431148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.431485 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.431448 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxl7\" (UniqueName: \"kubernetes.io/projected/779c70c3-62de-4d87-b035-1ac37190fb17-kube-api-access-xnxl7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6\" (UID: \"779c70c3-62de-4d87-b035-1ac37190fb17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.584357 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.584270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:47.679830 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.679797 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj"] Jul 30 03:49:47.684062 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.684040 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.687419 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.686687 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 03:49:47.687419 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.686762 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 03:49:47.687419 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.686698 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 03:49:47.687717 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.687663 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2s9s\"" Jul 30 03:49:47.696009 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.695943 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj"] Jul 30 03:49:47.722706 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722867 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722867 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722953 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e455cab-2974-4983-b37b-3e50b43a1711-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722953 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722953 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.722953 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.722941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlvv\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-kube-api-access-cqlvv\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.736708 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.735264 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6"] Jul 30 03:49:47.736708 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:49:47.735861 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779c70c3_62de_4d87_b035_1ac37190fb17.slice/crio-c1b0546ed7e547e8d8099d005e9b8764606e3454a56d9300add91d6081eabeba WatchSource:0}: Error finding container c1b0546ed7e547e8d8099d005e9b8764606e3454a56d9300add91d6081eabeba: Status 404 returned error can't find the container with id c1b0546ed7e547e8d8099d005e9b8764606e3454a56d9300add91d6081eabeba Jul 30 03:49:47.823665 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e455cab-2974-4983-b37b-3e50b43a1711-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823665 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823902 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823902 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlvv\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-kube-api-access-cqlvv\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823902 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823902 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.823902 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.823887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.824602 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.824579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.826179 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.826156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e455cab-2974-4983-b37b-3e50b43a1711-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.826272 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.826189 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.826573 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.826557 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.826615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.826604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e455cab-2974-4983-b37b-3e50b43a1711-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.834095 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.834074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlvv\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-kube-api-access-cqlvv\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.834273 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.834257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e455cab-2974-4983-b37b-3e50b43a1711-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-m2nrj\" (UID: \"2e455cab-2974-4983-b37b-3e50b43a1711\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:47.995704 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:47.995667 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" event={"ID":"779c70c3-62de-4d87-b035-1ac37190fb17","Type":"ContainerStarted","Data":"c1b0546ed7e547e8d8099d005e9b8764606e3454a56d9300add91d6081eabeba"} Jul 30 03:49:48.001848 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:48.001821 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:48.155190 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:48.155170 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj"] Jul 30 03:49:48.157256 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:49:48.157231 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e455cab_2974_4983_b37b_3e50b43a1711.slice/crio-bc5dfd76e19f4832632fa3e1a04051aa3ba63939d9892734e0738d984cd79ca4 WatchSource:0}: Error finding container bc5dfd76e19f4832632fa3e1a04051aa3ba63939d9892734e0738d984cd79ca4: Status 404 returned error can't find the container with id bc5dfd76e19f4832632fa3e1a04051aa3ba63939d9892734e0738d984cd79ca4 Jul 30 03:49:49.010001 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:49.009964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" event={"ID":"2e455cab-2974-4983-b37b-3e50b43a1711","Type":"ContainerStarted","Data":"bc5dfd76e19f4832632fa3e1a04051aa3ba63939d9892734e0738d984cd79ca4"} Jul 30 03:49:51.330214 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:51.330173 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:49:51.330534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:51.330249 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:49:51.330534 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:51.330279 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:49:51.330961 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:51.330937 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:49:51.331042 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:51.330990 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:49:52.027491 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.027420 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" event={"ID":"779c70c3-62de-4d87-b035-1ac37190fb17","Type":"ContainerStarted","Data":"1a67a62dc7260c7f8e5f7bf8b25a70556f0ccb3a81bbc4a4be7723ecb0159e17"} Jul 30 03:49:52.040007 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.039965 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" event={"ID":"2e455cab-2974-4983-b37b-3e50b43a1711","Type":"ContainerStarted","Data":"088218060d5a05538798c2063d2c634d3b81ebe207d097a2b020ef4771964326"} Jul 30 03:49:52.041384 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.041343 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:52.043610 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.043554 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-m2nrj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 03:49:52.043729 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.043672 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" podUID="2e455cab-2974-4983-b37b-3e50b43a1711" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 03:49:52.064452 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.064387 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" podStartSLOduration=1.473164587 podStartE2EDuration="5.064368676s" totalImagesPullingTime="3.591204089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:52.061678118 +0000 UTC m=+272.107177313" watchObservedRunningTime="2026-07-30 03:49:52.064368676 +0000 UTC m=+272.109867872" Jul 30 03:49:52.096721 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.096666 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" podStartSLOduration=1.925156736 podStartE2EDuration="5.096648043s" totalImagesPullingTime="3.171491307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:52.094910034 +0000 UTC m=+272.140409226" watchObservedRunningTime="2026-07-30 03:49:52.096648043 +0000 UTC m=+272.142147238" Jul 30 03:49:52.585424 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.585389 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:52.590649 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:52.590623 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:53.045535 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:53.045499 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:49:53.046089 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:53.046051 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-m2nrj" Jul 30 03:49:53.046378 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:49:53.046360 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6" Jul 30 03:50:20.370237 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:50:20.370204 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 03:52:15.750523 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.750484 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr"] Jul 30 03:52:15.754360 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.754339 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.756480 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.756445 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ddgxw\"" Jul 30 03:52:15.763174 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.763153 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr"] Jul 30 03:52:15.877322 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877334 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxcq\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-kube-api-access-fdxcq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877545 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877716 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877570 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877716 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877716 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.877716 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.877684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808a4069-437d-42e3-8dfc-8f612f5813bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978320 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978320 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808a4069-437d-42e3-8dfc-8f612f5813bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxcq\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-kube-api-access-fdxcq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978970 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978831 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.978970 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978909 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.979066 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.978996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.979102 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.979073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.979216 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.979200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808a4069-437d-42e3-8dfc-8f612f5813bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.980792 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.980768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.981028 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.981010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.986899 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.986875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:15.987105 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:15.987081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxcq\" (UniqueName: \"kubernetes.io/projected/808a4069-437d-42e3-8dfc-8f612f5813bf-kube-api-access-fdxcq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2tqpr\" (UID: \"808a4069-437d-42e3-8dfc-8f612f5813bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:16.066217 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.066117 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:16.209144 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.209119 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr"] Jul 30 03:52:16.211518 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:52:16.211484 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808a4069_437d_42e3_8dfc_8f612f5813bf.slice/crio-61642b60b44c66ccd46ec13c9edee2f7e751bbbcd73fca90b85c3b9763757666 WatchSource:0}: Error finding container 61642b60b44c66ccd46ec13c9edee2f7e751bbbcd73fca90b85c3b9763757666: Status 404 returned error can't find the container with id 61642b60b44c66ccd46ec13c9edee2f7e751bbbcd73fca90b85c3b9763757666 Jul 30 03:52:16.214016 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.213984 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:52:16.214102 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.214047 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:52:16.214102 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.214072 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:52:16.718215 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.718162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" event={"ID":"808a4069-437d-42e3-8dfc-8f612f5813bf","Type":"ContainerStarted","Data":"8619e25695f2d23460b8961965f59bf994eb41507a356ac3b64930b22784974a"} Jul 30 03:52:16.718405 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.718223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" event={"ID":"808a4069-437d-42e3-8dfc-8f612f5813bf","Type":"ContainerStarted","Data":"61642b60b44c66ccd46ec13c9edee2f7e751bbbcd73fca90b85c3b9763757666"} Jul 30 03:52:16.746892 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:16.746841 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" podStartSLOduration=1.746824744 podStartE2EDuration="1.746824744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:52:16.745394452 +0000 UTC m=+416.790893648" watchObservedRunningTime="2026-07-30 03:52:16.746824744 +0000 UTC m=+416.792323938" Jul 30 03:52:17.067297 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:17.067210 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:17.072542 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:17.072517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:17.722700 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:17.722661 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:52:17.723765 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:52:17.723748 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:53:08.805380 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.805342 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:53:08.809083 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.809061 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:08.812260 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.812237 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld5c2\"" Jul 30 03:53:08.812385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.812239 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 03:53:08.812385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.812239 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 03:53:08.817270 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.817247 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:53:08.991707 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.991668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxq9p\" (UniqueName: \"kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:08.991878 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:08.991715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.093285 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.093190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxq9p\" (UniqueName: \"kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.093285 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.093233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.096058 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.096033 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.100915 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.100893 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxq9p\" (UniqueName: \"kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p\") pod \"maas-controller-67944866c5-wkhs4\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.121909 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.121877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:09.255806 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.255720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:53:09.258190 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:53:09.258160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a97ecec_1ce8_4e21_9610_4dac77902220.slice/crio-9db8f2e60c981f11a89d8361b23998706e05a48840cdd4bc76e3ab09c538153a WatchSource:0}: Error finding container 9db8f2e60c981f11a89d8361b23998706e05a48840cdd4bc76e3ab09c538153a: Status 404 returned error can't find the container with id 9db8f2e60c981f11a89d8361b23998706e05a48840cdd4bc76e3ab09c538153a Jul 30 03:53:09.259496 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.259478 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 03:53:09.964992 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:09.964947 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-wkhs4" event={"ID":"4a97ecec-1ce8-4e21-9610-4dac77902220","Type":"ContainerStarted","Data":"9db8f2e60c981f11a89d8361b23998706e05a48840cdd4bc76e3ab09c538153a"} Jul 30 03:53:11.976718 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:11.976623 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-wkhs4" event={"ID":"4a97ecec-1ce8-4e21-9610-4dac77902220","Type":"ContainerStarted","Data":"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f"} Jul 30 03:53:11.977067 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:11.976737 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:12.014459 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:12.014403 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67944866c5-wkhs4" podStartSLOduration=1.602099734 podStartE2EDuration="4.014385803s" totalImagesPullingTime="2.412286069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:53:12.013363468 +0000 UTC m=+472.058862670" watchObservedRunningTime="2026-07-30 03:53:12.014385803 +0000 UTC m=+472.059884998" Jul 30 03:53:22.983140 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:22.983112 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:53:45.656889 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.656856 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-chpmj"] Jul 30 03:53:45.660157 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.660140 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.662969 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.662947 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m44qj\"" Jul 30 03:53:45.663103 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.662975 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 30 03:53:45.673339 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.673317 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-chpmj"] Jul 30 03:53:45.742280 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.742241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.742499 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.742439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwtw\" (UniqueName: \"kubernetes.io/projected/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-kube-api-access-lcwtw\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.843767 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.843727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwtw\" (UniqueName: \"kubernetes.io/projected/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-kube-api-access-lcwtw\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.843967 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.843807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.844599 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.844574 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.857333 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.857309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwtw\" (UniqueName: \"kubernetes.io/projected/77d3fddb-3d99-4314-a9c0-d0ea8fe860ef-kube-api-access-lcwtw\") pod \"payload-processing-5cc6cd444b-chpmj\" (UID: \"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef\") " pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:45.970347 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:45.970306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:46.117037 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:46.116998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-chpmj"] Jul 30 03:53:46.120345 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:53:46.120316 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d3fddb_3d99_4314_a9c0_d0ea8fe860ef.slice/crio-cdb070de9c4105122a3cfa788f857d7bd0067030959355560b676474465609a2 WatchSource:0}: Error finding container cdb070de9c4105122a3cfa788f857d7bd0067030959355560b676474465609a2: Status 404 returned error can't find the container with id cdb070de9c4105122a3cfa788f857d7bd0067030959355560b676474465609a2 Jul 30 03:53:46.143537 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:46.143504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" event={"ID":"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef","Type":"ContainerStarted","Data":"cdb070de9c4105122a3cfa788f857d7bd0067030959355560b676474465609a2"} Jul 30 03:53:48.174039 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:48.173944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" event={"ID":"77d3fddb-3d99-4314-a9c0-d0ea8fe860ef","Type":"ContainerStarted","Data":"9ab154a879aae96eadc00b1342583ec87e192634fe35ebfacd8e751db9f06daf"} Jul 30 03:53:48.174548 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:48.174044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:53:48.189647 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:48.189594 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" podStartSLOduration=1.221936292 podStartE2EDuration="3.189576328s" totalImagesPullingTime="1.967640036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:53:48.188577343 +0000 UTC m=+508.234076539" watchObservedRunningTime="2026-07-30 03:53:48.189576328 +0000 UTC m=+508.235075523" Jul 30 03:53:54.209134 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.209098 2568 generic.go:356] "Generic (PLEG): container finished" podID="808a4069-437d-42e3-8dfc-8f612f5813bf" containerID="8619e25695f2d23460b8961965f59bf994eb41507a356ac3b64930b22784974a" exitCode=0 Jul 30 03:53:54.209134 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.209137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" event={"ID":"808a4069-437d-42e3-8dfc-8f612f5813bf","Type":"ContainerDied","Data":"8619e25695f2d23460b8961965f59bf994eb41507a356ac3b64930b22784974a"} Jul 30 03:53:54.209615 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.209547 2568 scope.go:117] "RemoveContainer" containerID="8619e25695f2d23460b8961965f59bf994eb41507a356ac3b64930b22784974a" Jul 30 03:53:54.210635 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.210598 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:53:54.210745 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.210701 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:53:54.210786 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:54.210742 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 03:53:55.216314 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:55.216280 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" event={"ID":"808a4069-437d-42e3-8dfc-8f612f5813bf","Type":"ContainerStarted","Data":"22067c4b1c7bdaa8b048dff1683cb073c344b9a8852790dd7f060916b37ba976"} Jul 30 03:53:56.066852 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:56.066817 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:53:56.067224 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:56.067199 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2tqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 30 03:53:56.067412 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:56.067386 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" podUID="808a4069-437d-42e3-8dfc-8f612f5813bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 30 03:53:57.067372 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:57.067321 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2tqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 30 03:53:57.067935 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:57.067394 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" podUID="808a4069-437d-42e3-8dfc-8f612f5813bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 30 03:53:58.067500 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:58.067432 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2tqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 30 03:53:58.068088 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:58.068055 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" podUID="808a4069-437d-42e3-8dfc-8f612f5813bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 30 03:53:59.067100 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:59.066593 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2tqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 30 03:53:59.067100 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:59.066666 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" podUID="808a4069-437d-42e3-8dfc-8f612f5813bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 30 03:53:59.180664 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:53:59.180635 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-chpmj" Jul 30 03:54:00.071084 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:00.071043 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:54:00.071454 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:00.071431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:54:00.072063 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:00.072047 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2tqpr" Jul 30 03:54:18.837252 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.837170 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz"] Jul 30 03:54:18.840153 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.840130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.843267 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.843246 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 03:54:18.843385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.843284 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 03:54:18.843385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.843308 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bwnzj\"" Jul 30 03:54:18.843385 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.843346 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 03:54:18.851296 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.851268 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz"] Jul 30 03:54:18.990536 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.990707 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990543 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.990707 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.990707 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wvk\" (UniqueName: \"kubernetes.io/projected/b271df55-cf62-4b27-8540-0aef5074613a-kube-api-access-l5wvk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.990879 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:18.990879 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:18.990816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271df55-cf62-4b27-8540-0aef5074613a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092077 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.091976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092077 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wvk\" (UniqueName: \"kubernetes.io/projected/b271df55-cf62-4b27-8540-0aef5074613a-kube-api-access-l5wvk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092228 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271df55-cf62-4b27-8540-0aef5074613a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092305 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092275 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092453 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092453 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092418 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092690 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.092729 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.092718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.094516 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.094493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271df55-cf62-4b27-8540-0aef5074613a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.094742 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.094723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271df55-cf62-4b27-8540-0aef5074613a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.100120 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.100097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wvk\" (UniqueName: \"kubernetes.io/projected/b271df55-cf62-4b27-8540-0aef5074613a-kube-api-access-l5wvk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9fpgz\" (UID: \"b271df55-cf62-4b27-8540-0aef5074613a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.153716 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.153676 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:19.290810 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.290782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz"] Jul 30 03:54:19.293013 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:54:19.292986 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb271df55_cf62_4b27_8540_0aef5074613a.slice/crio-11d8b9fff4b726956f4f3ee5a52681653c256716130d7dddffe498755d66cefa WatchSource:0}: Error finding container 11d8b9fff4b726956f4f3ee5a52681653c256716130d7dddffe498755d66cefa: Status 404 returned error can't find the container with id 11d8b9fff4b726956f4f3ee5a52681653c256716130d7dddffe498755d66cefa Jul 30 03:54:19.336019 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:19.335987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" event={"ID":"b271df55-cf62-4b27-8540-0aef5074613a","Type":"ContainerStarted","Data":"11d8b9fff4b726956f4f3ee5a52681653c256716130d7dddffe498755d66cefa"} Jul 30 03:54:20.342141 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:20.342110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" event={"ID":"b271df55-cf62-4b27-8540-0aef5074613a","Type":"ContainerStarted","Data":"2e51c70e3fee87888eae169deefbb9a3c3cb9354ed9e2ace7d91c51397e77894"} Jul 30 03:54:20.364152 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:20.362172 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" podStartSLOduration=1.4299251499999999 podStartE2EDuration="2.362148493s" totalImagesPullingTime="932.223343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:54:20.360847843 +0000 UTC m=+540.406347069" watchObservedRunningTime="2026-07-30 03:54:20.362148493 +0000 UTC m=+540.407647693" Jul 30 03:54:22.945771 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:22.945739 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7"] Jul 30 03:54:22.948549 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:22.948527 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:22.950791 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:22.950770 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 30 03:54:22.958853 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:22.958829 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7"] Jul 30 03:54:23.028147 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.028321 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.028321 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dl24\" (UniqueName: \"kubernetes.io/projected/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-kube-api-access-6dl24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.028321 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.028321 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.028611 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.028385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129116 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dl24\" (UniqueName: \"kubernetes.io/projected/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-kube-api-access-6dl24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129359 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129665 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129570 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.129876 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.129854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.130487 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.130445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.132156 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.132133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.132894 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.132873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.136890 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.136871 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dl24\" (UniqueName: \"kubernetes.io/projected/e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc-kube-api-access-6dl24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4f7n7\" (UID: \"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.260822 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.260728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:23.394677 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:23.394652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7"] Jul 30 03:54:23.397096 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:54:23.397067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e7218c_ba1d_49c9_b0fb_dccfbc39ddcc.slice/crio-118a5313a89c0cd79f4c94b119fdda9bdaab97189380e7d8bd594e2fbccd02b3 WatchSource:0}: Error finding container 118a5313a89c0cd79f4c94b119fdda9bdaab97189380e7d8bd594e2fbccd02b3: Status 404 returned error can't find the container with id 118a5313a89c0cd79f4c94b119fdda9bdaab97189380e7d8bd594e2fbccd02b3 Jul 30 03:54:24.363928 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:24.363894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" event={"ID":"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc","Type":"ContainerStarted","Data":"8c94fd908f3c24970c507e708a9181846cd288bb7d8393251e631c8109268340"} Jul 30 03:54:24.364364 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:24.363953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" event={"ID":"e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc","Type":"ContainerStarted","Data":"118a5313a89c0cd79f4c94b119fdda9bdaab97189380e7d8bd594e2fbccd02b3"} Jul 30 03:54:24.379324 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:24.379277 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" podStartSLOduration=2.212673706 podStartE2EDuration="2.379260333s" totalImagesPullingTime="166.586627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:54:24.378446874 +0000 UTC m=+544.423946070" watchObservedRunningTime="2026-07-30 03:54:24.379260333 +0000 UTC m=+544.424759531" Jul 30 03:54:29.154227 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:29.154181 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:29.154675 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:29.154298 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:29.166956 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:29.166926 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:29.399431 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:29.399400 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9fpgz" Jul 30 03:54:33.260941 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:33.260900 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:33.261374 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:33.260968 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:33.273563 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:33.273537 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:54:33.426204 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:54:33.426169 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4f7n7" Jul 30 03:56:46.542253 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.542219 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:56:46.542933 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.542883 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67944866c5-wkhs4" podUID="4a97ecec-1ce8-4e21-9610-4dac77902220" containerName="manager" containerID="cri-o://7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f" gracePeriod=30 Jul 30 03:56:46.797797 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.797741 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:56:46.955697 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.955668 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert\") pod \"4a97ecec-1ce8-4e21-9610-4dac77902220\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " Jul 30 03:56:46.955879 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.955718 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxq9p\" (UniqueName: \"kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p\") pod \"4a97ecec-1ce8-4e21-9610-4dac77902220\" (UID: \"4a97ecec-1ce8-4e21-9610-4dac77902220\") " Jul 30 03:56:46.958134 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.958096 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4a97ecec-1ce8-4e21-9610-4dac77902220" (UID: "4a97ecec-1ce8-4e21-9610-4dac77902220"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:56:46.958134 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:46.958119 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p" (OuterVolumeSpecName: "kube-api-access-jxq9p") pod "4a97ecec-1ce8-4e21-9610-4dac77902220" (UID: "4a97ecec-1ce8-4e21-9610-4dac77902220"). InnerVolumeSpecName "kube-api-access-jxq9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:56:47.057517 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.057403 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a97ecec-1ce8-4e21-9610-4dac77902220-webhook-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:56:47.057517 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.057432 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxq9p\" (UniqueName: \"kubernetes.io/projected/4a97ecec-1ce8-4e21-9610-4dac77902220-kube-api-access-jxq9p\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:56:47.085479 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.085429 2568 generic.go:356] "Generic (PLEG): container finished" podID="4a97ecec-1ce8-4e21-9610-4dac77902220" containerID="7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f" exitCode=0 Jul 30 03:56:47.085651 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.085504 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-wkhs4" Jul 30 03:56:47.085651 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.085524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-wkhs4" event={"ID":"4a97ecec-1ce8-4e21-9610-4dac77902220","Type":"ContainerDied","Data":"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f"} Jul 30 03:56:47.085651 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.085588 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-wkhs4" event={"ID":"4a97ecec-1ce8-4e21-9610-4dac77902220","Type":"ContainerDied","Data":"9db8f2e60c981f11a89d8361b23998706e05a48840cdd4bc76e3ab09c538153a"} Jul 30 03:56:47.085651 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.085617 2568 scope.go:117] "RemoveContainer" containerID="7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f" Jul 30 03:56:47.101872 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.101853 2568 scope.go:117] "RemoveContainer" containerID="7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f" Jul 30 03:56:47.102165 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:56:47.102145 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f\": container with ID starting with 7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f not found: ID does not exist" containerID="7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f" Jul 30 03:56:47.102239 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.102174 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f"} err="failed to get container status \"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f\": rpc error: code = NotFound desc = could not find container \"7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f\": container with ID starting with 7a200ce253ae231343695dbbcb2dc4156653115b63be38205468d13158b6771f not found: ID does not exist" Jul 30 03:56:47.111835 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.111810 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:56:47.113441 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:47.113422 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67944866c5-wkhs4"] Jul 30 03:56:48.567365 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:56:48.567329 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a97ecec-1ce8-4e21-9610-4dac77902220" path="/var/lib/kubelet/pods/4a97ecec-1ce8-4e21-9610-4dac77902220/volumes" Jul 30 03:57:47.913231 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.913199 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:57:47.913669 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.913617 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a97ecec-1ce8-4e21-9610-4dac77902220" containerName="manager" Jul 30 03:57:47.913669 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.913629 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a97ecec-1ce8-4e21-9610-4dac77902220" containerName="manager" Jul 30 03:57:47.913750 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.913702 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a97ecec-1ce8-4e21-9610-4dac77902220" containerName="manager" Jul 30 03:57:47.916736 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.916721 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:47.919700 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.919679 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 03:57:47.919809 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.919707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld5c2\"" Jul 30 03:57:47.919809 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.919744 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 03:57:47.924108 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:47.924072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:57:48.072329 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.072294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.072329 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.072337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8cf\" (UniqueName: \"kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.173674 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.173643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.173838 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.173681 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8cf\" (UniqueName: \"kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.176323 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.176304 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.181900 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.181875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8cf\" (UniqueName: \"kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf\") pod \"maas-controller-67944866c5-xhmqw\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.229561 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.229526 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:48.378200 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:48.378150 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:57:48.382216 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:57:48.382177 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0361464_0b67_4054_acc0_a1b44f123a85.slice/crio-d8d8d416f469bfe9f93b618bc86266eb084f91a564592bc6d69ad5fb650b26d5 WatchSource:0}: Error finding container d8d8d416f469bfe9f93b618bc86266eb084f91a564592bc6d69ad5fb650b26d5: Status 404 returned error can't find the container with id d8d8d416f469bfe9f93b618bc86266eb084f91a564592bc6d69ad5fb650b26d5 Jul 30 03:57:49.392228 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:49.392196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-xhmqw" event={"ID":"a0361464-0b67-4054-acc0-a1b44f123a85","Type":"ContainerStarted","Data":"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd"} Jul 30 03:57:49.392644 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:49.392255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-xhmqw" event={"ID":"a0361464-0b67-4054-acc0-a1b44f123a85","Type":"ContainerStarted","Data":"d8d8d416f469bfe9f93b618bc86266eb084f91a564592bc6d69ad5fb650b26d5"} Jul 30 03:57:49.392644 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:49.392278 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:57:49.409202 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:57:49.409124 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67944866c5-xhmqw" podStartSLOduration=1.908199017 podStartE2EDuration="2.409103004s" totalImagesPullingTime="500.903987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:57:49.407247376 +0000 UTC m=+749.452746566" watchObservedRunningTime="2026-07-30 03:57:49.409103004 +0000 UTC m=+749.454602199" Jul 30 03:58:00.399573 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:00.399538 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:58:05.055865 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.055829 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:58:05.056303 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.056067 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67944866c5-xhmqw" podUID="a0361464-0b67-4054-acc0-a1b44f123a85" containerName="manager" containerID="cri-o://4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd" gracePeriod=30 Jul 30 03:58:05.310688 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.310621 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:58:05.442573 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.442531 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw8cf\" (UniqueName: \"kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf\") pod \"a0361464-0b67-4054-acc0-a1b44f123a85\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " Jul 30 03:58:05.442760 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.442599 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert\") pod \"a0361464-0b67-4054-acc0-a1b44f123a85\" (UID: \"a0361464-0b67-4054-acc0-a1b44f123a85\") " Jul 30 03:58:05.445623 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.445576 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a0361464-0b67-4054-acc0-a1b44f123a85" (UID: "a0361464-0b67-4054-acc0-a1b44f123a85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 03:58:05.445998 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.445952 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf" (OuterVolumeSpecName: "kube-api-access-rw8cf") pod "a0361464-0b67-4054-acc0-a1b44f123a85" (UID: "a0361464-0b67-4054-acc0-a1b44f123a85"). InnerVolumeSpecName "kube-api-access-rw8cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 03:58:05.473657 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.473618 2568 generic.go:356] "Generic (PLEG): container finished" podID="a0361464-0b67-4054-acc0-a1b44f123a85" containerID="4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd" exitCode=0 Jul 30 03:58:05.473863 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.473667 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-xhmqw" Jul 30 03:58:05.473863 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.473696 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-xhmqw" event={"ID":"a0361464-0b67-4054-acc0-a1b44f123a85","Type":"ContainerDied","Data":"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd"} Jul 30 03:58:05.473863 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.473744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-xhmqw" event={"ID":"a0361464-0b67-4054-acc0-a1b44f123a85","Type":"ContainerDied","Data":"d8d8d416f469bfe9f93b618bc86266eb084f91a564592bc6d69ad5fb650b26d5"} Jul 30 03:58:05.473863 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.473769 2568 scope.go:117] "RemoveContainer" containerID="4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd" Jul 30 03:58:05.486547 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.486524 2568 scope.go:117] "RemoveContainer" containerID="4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd" Jul 30 03:58:05.486909 ip-10-0-134-114 kubenswrapper[2568]: E0730 03:58:05.486888 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd\": container with ID starting with 4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd not found: ID does not exist" containerID="4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd" Jul 30 03:58:05.486988 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.486919 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd"} err="failed to get container status \"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd\": rpc error: code = NotFound desc = could not find container \"4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd\": container with ID starting with 4be05302f1e71edead4947f26ee4ef13086777bd83733415c820b6a9acf008dd not found: ID does not exist" Jul 30 03:58:05.520537 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.520499 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:58:05.524160 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.524125 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67944866c5-xhmqw"] Jul 30 03:58:05.544363 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.544319 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw8cf\" (UniqueName: \"kubernetes.io/projected/a0361464-0b67-4054-acc0-a1b44f123a85-kube-api-access-rw8cf\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:58:05.544363 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:05.544350 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0361464-0b67-4054-acc0-a1b44f123a85-webhook-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 03:58:06.290747 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.290714 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67944866c5-sjcwh"] Jul 30 03:58:06.291127 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.291114 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0361464-0b67-4054-acc0-a1b44f123a85" containerName="manager" Jul 30 03:58:06.291127 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.291125 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0361464-0b67-4054-acc0-a1b44f123a85" containerName="manager" Jul 30 03:58:06.291204 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.291187 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0361464-0b67-4054-acc0-a1b44f123a85" containerName="manager" Jul 30 03:58:06.295572 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.295549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.298590 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.298557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 03:58:06.298714 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.298558 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld5c2\"" Jul 30 03:58:06.298714 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.298558 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 03:58:06.303497 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.303452 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-sjcwh"] Jul 30 03:58:06.353161 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.353128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266v8\" (UniqueName: \"kubernetes.io/projected/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-kube-api-access-266v8\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.353342 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.353213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-webhook-cert\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.454184 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.454151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-webhook-cert\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.454405 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.454238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266v8\" (UniqueName: \"kubernetes.io/projected/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-kube-api-access-266v8\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.456826 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.456801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-webhook-cert\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.461998 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.461970 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-266v8\" (UniqueName: \"kubernetes.io/projected/e8e998a4-7071-4ed5-b2b4-8cb8c16e916d-kube-api-access-266v8\") pod \"maas-controller-67944866c5-sjcwh\" (UID: \"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d\") " pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.567572 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.567457 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0361464-0b67-4054-acc0-a1b44f123a85" path="/var/lib/kubelet/pods/a0361464-0b67-4054-acc0-a1b44f123a85/volumes" Jul 30 03:58:06.607688 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.607650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:06.744720 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:06.744695 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67944866c5-sjcwh"] Jul 30 03:58:06.747034 ip-10-0-134-114 kubenswrapper[2568]: W0730 03:58:06.747002 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e998a4_7071_4ed5_b2b4_8cb8c16e916d.slice/crio-8288ecb24d2e22872294f644938f1f8a26020ffb8513f4d7d9b6138b06653715 WatchSource:0}: Error finding container 8288ecb24d2e22872294f644938f1f8a26020ffb8513f4d7d9b6138b06653715: Status 404 returned error can't find the container with id 8288ecb24d2e22872294f644938f1f8a26020ffb8513f4d7d9b6138b06653715 Jul 30 03:58:07.486623 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:07.486583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-sjcwh" event={"ID":"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d","Type":"ContainerStarted","Data":"28b77d38857d2d04f2c894f507ba9fdb652a681b88c54fba37b138f8f4a778ab"} Jul 30 03:58:07.487054 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:07.486637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67944866c5-sjcwh" event={"ID":"e8e998a4-7071-4ed5-b2b4-8cb8c16e916d","Type":"ContainerStarted","Data":"8288ecb24d2e22872294f644938f1f8a26020ffb8513f4d7d9b6138b06653715"} Jul 30 03:58:07.487054 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:07.486700 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 03:58:07.504175 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:07.504121 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67944866c5-sjcwh" podStartSLOduration=1.183429775 podStartE2EDuration="1.504104308s" totalImagesPullingTime="320.674533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:58:07.502178949 +0000 UTC m=+767.547678144" watchObservedRunningTime="2026-07-30 03:58:07.504104308 +0000 UTC m=+767.549603502" Jul 30 03:58:18.493423 ip-10-0-134-114 kubenswrapper[2568]: I0730 03:58:18.493392 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67944866c5-sjcwh" Jul 30 04:00:00.141995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.141963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl"] Jul 30 04:00:00.145443 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.145426 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" Jul 30 04:00:00.147546 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.147522 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:00:00.154839 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.154819 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" Jul 30 04:00:00.160669 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.160642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl"] Jul 30 04:00:00.292036 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.291826 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl"] Jul 30 04:00:00.294496 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:00:00.294446 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5ceb01_9140_4025_be03_0ad8568e95e8.slice/crio-854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec WatchSource:0}: Error finding container 854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec: Status 404 returned error can't find the container with id 854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec Jul 30 04:00:00.297019 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:00.297003 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 04:00:01.052323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:01.052285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" event={"ID":"6e5ceb01-9140-4025-be03-0ad8568e95e8","Type":"ContainerStarted","Data":"854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec"} Jul 30 04:00:03.066136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:03.066100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" event={"ID":"6e5ceb01-9140-4025-be03-0ad8568e95e8","Type":"ContainerStarted","Data":"156afb065eb958f96843ad06b67c69dd99b0364daceb1734be0d064d69d2b615"} Jul 30 04:00:03.082819 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:03.082764 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" podStartSLOduration=0.702615594 podStartE2EDuration="3.082746422s" totalImagesPullingTime="2.380130828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:00:03.08136155 +0000 UTC m=+883.126860746" watchObservedRunningTime="2026-07-30 04:00:03.082746422 +0000 UTC m=+883.128245616" Jul 30 04:00:04.074557 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:04.074514 2568 generic.go:356] "Generic (PLEG): container finished" podID="6e5ceb01-9140-4025-be03-0ad8568e95e8" containerID="156afb065eb958f96843ad06b67c69dd99b0364daceb1734be0d064d69d2b615" exitCode=0 Jul 30 04:00:04.074557 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:04.074560 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" event={"ID":"6e5ceb01-9140-4025-be03-0ad8568e95e8","Type":"ContainerDied","Data":"156afb065eb958f96843ad06b67c69dd99b0364daceb1734be0d064d69d2b615"} Jul 30 04:00:05.222618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:05.222586 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" Jul 30 04:00:06.085625 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:06.085583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" event={"ID":"6e5ceb01-9140-4025-be03-0ad8568e95e8","Type":"ContainerDied","Data":"854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec"} Jul 30 04:00:06.085625 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:06.085607 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl" Jul 30 04:00:06.085847 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:00:06.085643 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854774d00c70209faab408913edc39e90c53ae820a0b32877bd5648d705846ec" Jul 30 04:15:00.139953 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.139918 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv"] Jul 30 04:15:00.140431 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.140331 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5ceb01-9140-4025-be03-0ad8568e95e8" containerName="cleanup" Jul 30 04:15:00.140431 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.140342 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5ceb01-9140-4025-be03-0ad8568e95e8" containerName="cleanup" Jul 30 04:15:00.140431 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.140422 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5ceb01-9140-4025-be03-0ad8568e95e8" containerName="cleanup" Jul 30 04:15:00.143372 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.143356 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" Jul 30 04:15:00.145630 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.145615 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:15:00.152853 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.152837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" Jul 30 04:15:00.159551 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.159528 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv"] Jul 30 04:15:00.289835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.289630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv"] Jul 30 04:15:00.292214 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:15:00.292179 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbbe69f_d093_4a95_b828_4010e6e0d884.slice/crio-1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09 WatchSource:0}: Error finding container 1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09: Status 404 returned error can't find the container with id 1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09 Jul 30 04:15:00.525533 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.525496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" event={"ID":"efbbe69f-d093-4a95-b828-4010e6e0d884","Type":"ContainerStarted","Data":"1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09"} Jul 30 04:15:00.542670 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.542636 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:00.547770 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.547746 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.549920 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.549899 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5a036514-openshift-default-dockercfg-fbttw\"" Jul 30 04:15:00.570280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570411 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570411 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570586 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8f5\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570643 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570643 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570734 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570771 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.570814 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.570774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.573238 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.573212 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:00.671481 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671680 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671680 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671680 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671680 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671680 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8f5\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.671939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.671809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.672639 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.672102 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.672639 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.672128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.672639 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.672308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.672639 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.672334 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.672639 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.672450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.674075 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.674053 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.674186 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.674168 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.678679 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.678661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.678910 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.678893 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8f5\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5\") pod \"e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:00.864712 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:00.864622 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:01.012089 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.012066 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:01.014897 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:15:01.014870 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb774c2c7_faca_42af_96aa_c604b0c8b6c7.slice/crio-ffc6f0be5d4599f92262e852630b964bc868ca57632d8697dbf63c5f0e785a17 WatchSource:0}: Error finding container ffc6f0be5d4599f92262e852630b964bc868ca57632d8697dbf63c5f0e785a17: Status 404 returned error can't find the container with id ffc6f0be5d4599f92262e852630b964bc868ca57632d8697dbf63c5f0e785a17 Jul 30 04:15:01.017059 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.017022 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:01.017146 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.017092 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:01.017146 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.017128 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:01.532612 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.532516 2568 generic.go:356] "Generic (PLEG): container finished" podID="efbbe69f-d093-4a95-b828-4010e6e0d884" containerID="27be596a9582e1428c19a68178ea526eeddd58009aacbe8da7af2662facc1335" exitCode=0 Jul 30 04:15:01.533007 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.532607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" event={"ID":"efbbe69f-d093-4a95-b828-4010e6e0d884","Type":"ContainerDied","Data":"27be596a9582e1428c19a68178ea526eeddd58009aacbe8da7af2662facc1335"} Jul 30 04:15:01.534156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.534134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerStarted","Data":"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce"} Jul 30 04:15:01.534267 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.534181 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerStarted","Data":"ffc6f0be5d4599f92262e852630b964bc868ca57632d8697dbf63c5f0e785a17"} Jul 30 04:15:01.563554 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.563503 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podStartSLOduration=1.5634841449999999 podStartE2EDuration="1.563484145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:01.561901223 +0000 UTC m=+1781.607400417" watchObservedRunningTime="2026-07-30 04:15:01.563484145 +0000 UTC m=+1781.608983330" Jul 30 04:15:01.865603 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:01.865513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:02.680280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:02.680252 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" Jul 30 04:15:02.870654 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:02.870548 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:03.545963 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:03.545928 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" Jul 30 04:15:03.546173 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:03.545971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9cdbv" event={"ID":"efbbe69f-d093-4a95-b828-4010e6e0d884","Type":"ContainerDied","Data":"1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09"} Jul 30 04:15:03.546173 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:03.546037 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3aa0d21b1b8e5a6d6d8515f9145bf92e96a7bb8c8010c292b8227c82da1f09" Jul 30 04:15:03.546355 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:03.546338 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:03.547211 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:03.547193 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:04.781485 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.781427 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:04.786905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.786876 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbbe69f-d093-4a95-b828-4010e6e0d884" containerName="cleanup" Jul 30 04:15:04.786905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.786900 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbbe69f-d093-4a95-b828-4010e6e0d884" containerName="cleanup" Jul 30 04:15:04.787105 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.786968 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="efbbe69f-d093-4a95-b828-4010e6e0d884" containerName="cleanup" Jul 30 04:15:04.790123 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.790107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:04.792595 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.792570 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5a036514\"" Jul 30 04:15:04.792738 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.792579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5a036514-dockercfg-pwpx4\"" Jul 30 04:15:04.794285 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.794259 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:04.917937 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.917900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:04.918146 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:04.917946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs29\" (UniqueName: \"kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.019687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.019650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.019687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.019690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs29\" (UniqueName: \"kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.020403 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.020377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.027259 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.027226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs29\" (UniqueName: \"kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29\") pod \"payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.102530 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.102412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.245383 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.245359 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:05.247859 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:15:05.247821 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbfc309_d55e_4523_a4d4_18680942a1af.slice/crio-ceb67ebe2193aa43f23ea965238ac7b95a2fad02b8c13712ab533ff9ca48f077 WatchSource:0}: Error finding container ceb67ebe2193aa43f23ea965238ac7b95a2fad02b8c13712ab533ff9ca48f077: Status 404 returned error can't find the container with id ceb67ebe2193aa43f23ea965238ac7b95a2fad02b8c13712ab533ff9ca48f077 Jul 30 04:15:05.558936 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.558900 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" event={"ID":"5dbfc309-d55e-4523-a4d4-18680942a1af","Type":"ContainerStarted","Data":"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d"} Jul 30 04:15:05.559173 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.558968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" event={"ID":"5dbfc309-d55e-4523-a4d4-18680942a1af","Type":"ContainerStarted","Data":"ceb67ebe2193aa43f23ea965238ac7b95a2fad02b8c13712ab533ff9ca48f077"} Jul 30 04:15:05.559173 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.558995 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:05.575058 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:05.574998 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" podStartSLOduration=1.574982296 podStartE2EDuration="1.574982296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:05.572948228 +0000 UTC m=+1785.618447424" watchObservedRunningTime="2026-07-30 04:15:05.574982296 +0000 UTC m=+1785.620481490" Jul 30 04:15:16.568962 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:16.568927 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:17.626176 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.626141 2568 generic.go:356] "Generic (PLEG): container finished" podID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerID="b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce" exitCode=0 Jul 30 04:15:17.626658 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.626233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerDied","Data":"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce"} Jul 30 04:15:17.626737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.626707 2568 scope.go:117] "RemoveContainer" containerID="b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce" Jul 30 04:15:17.627779 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.627748 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:17.627867 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.627824 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:17.627867 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:17.627861 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:18.546823 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:18.546768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:18.633092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:18.633060 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerStarted","Data":"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc"} Jul 30 04:15:18.869592 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:18.868641 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:18.871489 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:18.870091 2568 patch_prober.go:28] interesting pod/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 30 04:15:18.871489 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:18.870144 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 30 04:15:19.865833 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:19.865323 2568 patch_prober.go:28] interesting pod/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 30 04:15:19.865833 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:19.865400 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 30 04:15:20.865633 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:20.865587 2568 patch_prober.go:28] interesting pod/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 30 04:15:20.865858 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:20.865679 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 30 04:15:21.866090 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:21.866041 2568 patch_prober.go:28] interesting pod/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 30 04:15:21.866644 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:21.866130 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 30 04:15:22.870111 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:22.870077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:22.870594 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:22.870496 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:22.871138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:22.871117 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:28.519380 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.519338 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt"] Jul 30 04:15:28.522940 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.522920 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:28.525823 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.525801 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:15:28.526164 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.525830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:15:28.528160 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.528027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt"] Jul 30 04:15:28.571926 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.571884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5a036514-qkldt\" (UID: \"572ec798-94c8-42d6-bc1f-157790972e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:28.673351 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.673322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5a036514-qkldt\" (UID: \"572ec798-94c8-42d6-bc1f-157790972e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:28.673928 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.673909 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5a036514-qkldt\" (UID: \"572ec798-94c8-42d6-bc1f-157790972e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:28.836187 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.836105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:28.976087 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:28.976063 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt"] Jul 30 04:15:28.978663 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:15:28.978632 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572ec798_94c8_42d6_bc1f_157790972e84.slice/crio-43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272 WatchSource:0}: Error finding container 43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272: Status 404 returned error can't find the container with id 43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272 Jul 30 04:15:29.691743 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:29.691709 2568 generic.go:356] "Generic (PLEG): container finished" podID="572ec798-94c8-42d6-bc1f-157790972e84" containerID="5beb9325058338e98c15e607018452527bf7cb4cedb493bdc8ae84083aae844c" exitCode=0 Jul 30 04:15:29.692102 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:29.691794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" event={"ID":"572ec798-94c8-42d6-bc1f-157790972e84","Type":"ContainerDied","Data":"5beb9325058338e98c15e607018452527bf7cb4cedb493bdc8ae84083aae844c"} Jul 30 04:15:29.692102 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:29.691855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" event={"ID":"572ec798-94c8-42d6-bc1f-157790972e84","Type":"ContainerStarted","Data":"43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272"} Jul 30 04:15:30.833447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:30.833424 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:30.896335 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:30.896291 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca\") pod \"572ec798-94c8-42d6-bc1f-157790972e84\" (UID: \"572ec798-94c8-42d6-bc1f-157790972e84\") " Jul 30 04:15:30.896751 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:30.896728 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "572ec798-94c8-42d6-bc1f-157790972e84" (UID: "572ec798-94c8-42d6-bc1f-157790972e84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:15:30.998117 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:30.998026 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572ec798-94c8-42d6-bc1f-157790972e84-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:31.703742 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:31.703712 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" Jul 30 04:15:31.704028 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:31.703744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt" event={"ID":"572ec798-94c8-42d6-bc1f-157790972e84","Type":"ContainerDied","Data":"43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272"} Jul 30 04:15:31.704028 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:31.703792 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43cde3133faf5fabf2608a3be3707cc0c95c06e42263ff9f6eec7befb50a1272" Jul 30 04:15:32.603868 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.603836 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:32.604352 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.604094 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" podUID="5dbfc309-d55e-4523-a4d4-18680942a1af" containerName="payload-pre-processing" containerID="cri-o://b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d" gracePeriod=30 Jul 30 04:15:32.733270 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.733192 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt"] Jul 30 04:15:32.736025 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.735998 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5a036514-qkldt"] Jul 30 04:15:32.877744 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.877708 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:32.877992 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.877966 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" containerID="cri-o://ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc" gracePeriod=30 Jul 30 04:15:32.889660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.889639 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:32.918158 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.918117 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume\") pod \"5dbfc309-d55e-4523-a4d4-18680942a1af\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " Jul 30 04:15:32.918356 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.918261 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cs29\" (UniqueName: \"kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29\") pod \"5dbfc309-d55e-4523-a4d4-18680942a1af\" (UID: \"5dbfc309-d55e-4523-a4d4-18680942a1af\") " Jul 30 04:15:32.918507 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.918456 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5dbfc309-d55e-4523-a4d4-18680942a1af" (UID: "5dbfc309-d55e-4523-a4d4-18680942a1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:15:32.920688 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:32.920656 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29" (OuterVolumeSpecName: "kube-api-access-4cs29") pod "5dbfc309-d55e-4523-a4d4-18680942a1af" (UID: "5dbfc309-d55e-4523-a4d4-18680942a1af"). InnerVolumeSpecName "kube-api-access-4cs29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:15:33.019585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.019551 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cs29\" (UniqueName: \"kubernetes.io/projected/5dbfc309-d55e-4523-a4d4-18680942a1af-kube-api-access-4cs29\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:33.019585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.019577 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbfc309-d55e-4523-a4d4-18680942a1af-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:33.717095 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.717060 2568 generic.go:356] "Generic (PLEG): container finished" podID="5dbfc309-d55e-4523-a4d4-18680942a1af" containerID="b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d" exitCode=0 Jul 30 04:15:33.717530 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.717099 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" Jul 30 04:15:33.717530 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.717113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" event={"ID":"5dbfc309-d55e-4523-a4d4-18680942a1af","Type":"ContainerDied","Data":"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d"} Jul 30 04:15:33.717530 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.717161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d" event={"ID":"5dbfc309-d55e-4523-a4d4-18680942a1af","Type":"ContainerDied","Data":"ceb67ebe2193aa43f23ea965238ac7b95a2fad02b8c13712ab533ff9ca48f077"} Jul 30 04:15:33.717530 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.717184 2568 scope.go:117] "RemoveContainer" containerID="b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d" Jul 30 04:15:33.726562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.726544 2568 scope.go:117] "RemoveContainer" containerID="b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d" Jul 30 04:15:33.726832 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:15:33.726813 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d\": container with ID starting with b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d not found: ID does not exist" containerID="b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d" Jul 30 04:15:33.726893 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.726840 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d"} err="failed to get container status \"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d\": rpc error: code = NotFound desc = could not find container \"b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d\": container with ID starting with b857e38d0f1c8a8905bd685be6b8bcf744f92506ae4283b6141bdeffa40a158d not found: ID does not exist" Jul 30 04:15:33.738029 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.737996 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:33.742039 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.742010 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5a036514-79d9788ddd-2q68d"] Jul 30 04:15:33.892879 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.892846 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:15:33.893428 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893401 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbfc309-d55e-4523-a4d4-18680942a1af" containerName="payload-pre-processing" Jul 30 04:15:33.893590 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893457 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbfc309-d55e-4523-a4d4-18680942a1af" containerName="payload-pre-processing" Jul 30 04:15:33.893590 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893493 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572ec798-94c8-42d6-bc1f-157790972e84" containerName="revoke-keys" Jul 30 04:15:33.893590 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893501 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="572ec798-94c8-42d6-bc1f-157790972e84" containerName="revoke-keys" Jul 30 04:15:33.893756 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893625 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbfc309-d55e-4523-a4d4-18680942a1af" containerName="payload-pre-processing" Jul 30 04:15:33.893756 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.893639 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="572ec798-94c8-42d6-bc1f-157790972e84" containerName="revoke-keys" Jul 30 04:15:33.897122 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.897100 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.899331 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.899299 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-974e458f-gw-openshift-default-dockercfg-twshn\"" Jul 30 04:15:33.908260 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.908239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:15:33.927681 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.927681 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.927945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927709 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.927945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.927945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.927945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.928171 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927956 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.928171 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.927983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wmt\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:33.928171 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:33.928020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.028939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.028853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.028939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.028915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.028991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wmt\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029499 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029499 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029499 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029434 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029677 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029734 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029682 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.029734 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.029687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.031563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.031541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.031677 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.031662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.041673 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.041643 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.041870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.041851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wmt\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt\") pod \"e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.208778 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.208737 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:34.346312 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.346287 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:15:34.348519 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:15:34.348492 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6b0599_e37d_4795_afed_f89ee2c43708.slice/crio-a901fd62417fb425e8c853047fae7028fb3572ccb6d2738bb816bee7d1791f58 WatchSource:0}: Error finding container a901fd62417fb425e8c853047fae7028fb3572ccb6d2738bb816bee7d1791f58: Status 404 returned error can't find the container with id a901fd62417fb425e8c853047fae7028fb3572ccb6d2738bb816bee7d1791f58 Jul 30 04:15:34.350602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.350570 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:34.350703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.350634 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:34.350703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.350668 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:34.569250 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.569162 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572ec798-94c8-42d6-bc1f-157790972e84" path="/var/lib/kubelet/pods/572ec798-94c8-42d6-bc1f-157790972e84/volumes" Jul 30 04:15:34.569611 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.569596 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbfc309-d55e-4523-a4d4-18680942a1af" path="/var/lib/kubelet/pods/5dbfc309-d55e-4523-a4d4-18680942a1af/volumes" Jul 30 04:15:34.724542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.724505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerStarted","Data":"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c"} Jul 30 04:15:34.724997 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.724571 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerStarted","Data":"a901fd62417fb425e8c853047fae7028fb3572ccb6d2738bb816bee7d1791f58"} Jul 30 04:15:34.748439 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:34.748385 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podStartSLOduration=1.748369555 podStartE2EDuration="1.748369555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:34.745408474 +0000 UTC m=+1814.790907668" watchObservedRunningTime="2026-07-30 04:15:34.748369555 +0000 UTC m=+1814.793868750" Jul 30 04:15:35.209708 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:35.209674 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:35.214648 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:35.214624 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:35.730497 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:35.730442 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:35.731428 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:35.731407 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:37.873053 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:37.873016 2568 patch_prober.go:28] interesting pod/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:15:37.873591 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:37.873085 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:15:38.135865 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.135799 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:38.173679 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173641 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v8f5\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.173888 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173742 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.173888 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173796 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.173888 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173820 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.173888 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173855 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.174139 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173928 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.174139 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.173975 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.174139 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174019 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.174139 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174057 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data\") pod \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\" (UID: \"b774c2c7-faca-42af-96aa-c604b0c8b6c7\") " Jul 30 04:15:38.174707 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:15:38.174707 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174662 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:15:38.175148 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:15:38.175148 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.174983 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:15:38.175335 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.175268 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:15:38.176209 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.176182 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5" (OuterVolumeSpecName: "kube-api-access-5v8f5") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "kube-api-access-5v8f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:15:38.176672 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.176649 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:15:38.176915 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.176893 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:15:38.177110 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.177095 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "b774c2c7-faca-42af-96aa-c604b0c8b6c7" (UID: "b774c2c7-faca-42af-96aa-c604b0c8b6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:15:38.275958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275915 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.275958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275951 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.275958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275962 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.275958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275970 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.276234 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275978 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8f5\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-kube-api-access-5v8f5\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.276234 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275986 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.276234 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.275994 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.276234 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.276002 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b774c2c7-faca-42af-96aa-c604b0c8b6c7-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.276234 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.276010 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b774c2c7-faca-42af-96aa-c604b0c8b6c7-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:15:38.748568 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.748538 2568 generic.go:356] "Generic (PLEG): container finished" podID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerID="ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc" exitCode=0 Jul 30 04:15:38.748747 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.748587 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" Jul 30 04:15:38.748747 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.748622 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerDied","Data":"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc"} Jul 30 04:15:38.748747 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.748672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6" event={"ID":"b774c2c7-faca-42af-96aa-c604b0c8b6c7","Type":"ContainerDied","Data":"ffc6f0be5d4599f92262e852630b964bc868ca57632d8697dbf63c5f0e785a17"} Jul 30 04:15:38.748747 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.748693 2568 scope.go:117] "RemoveContainer" containerID="ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc" Jul 30 04:15:38.757717 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.757702 2568 scope.go:117] "RemoveContainer" containerID="b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce" Jul 30 04:15:38.765503 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.765479 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:38.766739 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.766708 2568 scope.go:117] "RemoveContainer" containerID="ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc" Jul 30 04:15:38.767361 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:15:38.767330 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc\": container with ID starting with ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc not found: ID does not exist" containerID="ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc" Jul 30 04:15:38.767498 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.767371 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc"} err="failed to get container status \"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc\": rpc error: code = NotFound desc = could not find container \"ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc\": container with ID starting with ede5223df401a3c60ecd8bf83b2efe5ec7e11beece2cca6f523202030e56bdcc not found: ID does not exist" Jul 30 04:15:38.767498 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.767399 2568 scope.go:117] "RemoveContainer" containerID="b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce" Jul 30 04:15:38.767770 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:15:38.767751 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce\": container with ID starting with b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce not found: ID does not exist" containerID="b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce" Jul 30 04:15:38.767817 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.767776 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce"} err="failed to get container status \"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce\": rpc error: code = NotFound desc = could not find container \"b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce\": container with ID starting with b5a43cb63469be7f5fd2843e284294727e008e4059f4b07ee1fc702b3a0d12ce not found: ID does not exist" Jul 30 04:15:38.769244 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:38.769224 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5a036514-openshift-default-654f6946c8-w7lr6"] Jul 30 04:15:40.567654 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:40.567617 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" path="/var/lib/kubelet/pods/b774c2c7-faca-42af-96aa-c604b0c8b6c7/volumes" Jul 30 04:15:50.732202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.732167 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 30 04:15:50.732613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.732239 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 30 04:15:50.812625 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.812546 2568 generic.go:356] "Generic (PLEG): container finished" podID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerID="edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c" exitCode=0 Jul 30 04:15:50.812764 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.812625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerDied","Data":"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c"} Jul 30 04:15:50.813071 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.813055 2568 scope.go:117] "RemoveContainer" containerID="edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c" Jul 30 04:15:50.814103 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.814072 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:50.814217 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.814159 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:50.814290 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:50.814223 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:15:51.819726 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:51.819685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerStarted","Data":"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad"} Jul 30 04:15:52.209188 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:52.209146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:52.209437 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:52.209417 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 30 04:15:52.209522 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:52.209494 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 30 04:15:53.209410 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:53.209370 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 30 04:15:53.209839 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:53.209438 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 30 04:15:54.209712 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:54.209671 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 30 04:15:54.210106 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:54.209735 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 30 04:15:55.209374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:55.209338 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 30 04:15:55.209586 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:55.209409 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 30 04:15:56.213085 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:56.213053 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:56.213549 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:56.213325 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:15:56.214064 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:15:56.214046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:16:01.255702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.255663 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl"] Jul 30 04:16:01.256195 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.256178 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:01.256247 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.256197 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:01.256283 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.256273 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:01.256316 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.256283 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:01.258918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.258901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:01.261078 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.261054 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:16:01.261210 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.261099 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:16:01.266136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.266107 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl"] Jul 30 04:16:01.323162 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.323128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-974e458f-nczkl\" (UID: \"445e3503-402d-4e3e-af8a-faaeb6762d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:01.424635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.424596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-974e458f-nczkl\" (UID: \"445e3503-402d-4e3e-af8a-faaeb6762d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:01.425251 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.425229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-974e458f-nczkl\" (UID: \"445e3503-402d-4e3e-af8a-faaeb6762d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:01.571873 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.571776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:01.739509 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.739477 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl"] Jul 30 04:16:01.741694 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:16:01.741666 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445e3503_402d_4e3e_af8a_faaeb6762d92.slice/crio-a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c WatchSource:0}: Error finding container a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c: Status 404 returned error can't find the container with id a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c Jul 30 04:16:01.872199 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:01.872088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" event={"ID":"445e3503-402d-4e3e-af8a-faaeb6762d92","Type":"ContainerStarted","Data":"a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c"} Jul 30 04:16:02.879526 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:02.879416 2568 generic.go:356] "Generic (PLEG): container finished" podID="445e3503-402d-4e3e-af8a-faaeb6762d92" containerID="9ddd118f8703724872ff777715679b290e83ab5776813891f131fc8bab0744e3" exitCode=0 Jul 30 04:16:02.879526 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:02.879505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" event={"ID":"445e3503-402d-4e3e-af8a-faaeb6762d92","Type":"ContainerDied","Data":"9ddd118f8703724872ff777715679b290e83ab5776813891f131fc8bab0744e3"} Jul 30 04:16:04.023762 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.023732 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:04.050567 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.050535 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca\") pod \"445e3503-402d-4e3e-af8a-faaeb6762d92\" (UID: \"445e3503-402d-4e3e-af8a-faaeb6762d92\") " Jul 30 04:16:04.050869 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.050841 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "445e3503-402d-4e3e-af8a-faaeb6762d92" (UID: "445e3503-402d-4e3e-af8a-faaeb6762d92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:16:04.152109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.152014 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e3503-402d-4e3e-af8a-faaeb6762d92-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:04.890640 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.890560 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" Jul 30 04:16:04.890640 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.890587 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl" event={"ID":"445e3503-402d-4e3e-af8a-faaeb6762d92","Type":"ContainerDied","Data":"a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c"} Jul 30 04:16:04.890640 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:04.890632 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a85062ef52a407e1abc12176816de69efd5542af9346ff900c8ed64ca11d874c" Jul 30 04:16:07.125155 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:07.125114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl"] Jul 30 04:16:07.129901 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:07.129875 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-974e458f-nczkl"] Jul 30 04:16:07.270552 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:07.270518 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:16:07.270811 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:07.270765 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" containerID="cri-o://32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad" gracePeriod=30 Jul 30 04:16:08.026141 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026104 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:08.026624 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026601 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445e3503-402d-4e3e-af8a-faaeb6762d92" containerName="revoke-keys" Jul 30 04:16:08.026782 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026764 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="445e3503-402d-4e3e-af8a-faaeb6762d92" containerName="revoke-keys" Jul 30 04:16:08.026866 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026832 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:08.026866 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026845 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b774c2c7-faca-42af-96aa-c604b0c8b6c7" containerName="istio-proxy" Jul 30 04:16:08.026988 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.026975 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="445e3503-402d-4e3e-af8a-faaeb6762d92" containerName="revoke-keys" Jul 30 04:16:08.029356 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.029336 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.031803 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.031782 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b51d8f16-openshift-default-dockercfg-9z8z9\"" Jul 30 04:16:08.041395 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.041365 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:08.089943 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.089909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.089970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.089992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89th\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.090354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.090226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.191715 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89th\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192156 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.191960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.192282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.192516 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.192549 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.192574 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.192613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.192594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.194995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.194969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.195150 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.195134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.199221 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.199197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89th\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.199630 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.199612 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token\") pod \"e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.340121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.340022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:08.481784 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.481759 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:08.484035 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:16:08.483999 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34790bc1_27fc_465d_a56e_dcadc354ea81.slice/crio-29a148e806b6f1ef58d9e9b7d342db7f295b09708da240a634afd8e807fe3907 WatchSource:0}: Error finding container 29a148e806b6f1ef58d9e9b7d342db7f295b09708da240a634afd8e807fe3907: Status 404 returned error can't find the container with id 29a148e806b6f1ef58d9e9b7d342db7f295b09708da240a634afd8e807fe3907 Jul 30 04:16:08.486233 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.486201 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:08.486310 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.486267 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:08.486310 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.486293 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:08.568207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.568174 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="445e3503-402d-4e3e-af8a-faaeb6762d92" path="/var/lib/kubelet/pods/445e3503-402d-4e3e-af8a-faaeb6762d92/volumes" Jul 30 04:16:08.913776 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.913737 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerStarted","Data":"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251"} Jul 30 04:16:08.913934 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.913796 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerStarted","Data":"29a148e806b6f1ef58d9e9b7d342db7f295b09708da240a634afd8e807fe3907"} Jul 30 04:16:08.931574 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:08.931515 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podStartSLOduration=0.93149518 podStartE2EDuration="931.49518ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:16:08.930928009 +0000 UTC m=+1848.976427205" watchObservedRunningTime="2026-07-30 04:16:08.93149518 +0000 UTC m=+1848.976994366" Jul 30 04:16:09.341021 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:09.340982 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:09.345927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:09.345899 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:09.918982 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:09.918950 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:09.919978 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:09.919957 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:11.216354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:11.216315 2568 patch_prober.go:28] interesting pod/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:16:11.216787 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:11.216390 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:16:12.160660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.160622 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:12.163579 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.163561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.165718 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.165693 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b51d8f16\"" Jul 30 04:16:12.165848 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.165779 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b51d8f16-dockercfg-d9qln\"" Jul 30 04:16:12.173871 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.173844 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:12.236532 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.236495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tzr\" (UniqueName: \"kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.236532 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.236534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.337705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.337666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tzr\" (UniqueName: \"kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.337705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.337707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.338374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.338355 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.344780 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.344758 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tzr\" (UniqueName: \"kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr\") pod \"payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.475977 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.475942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.525275 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.525243 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:16:12.620584 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.620565 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:12.622910 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:16:12.622882 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d9720e_5a18_46b2_a23b_6212034fe22f.slice/crio-d0489a31fb3e3c3ded178e8953eada75f853f3242274af03db7791bc4f9beef8 WatchSource:0}: Error finding container d0489a31fb3e3c3ded178e8953eada75f853f3242274af03db7791bc4f9beef8: Status 404 returned error can't find the container with id d0489a31fb3e3c3ded178e8953eada75f853f3242274af03db7791bc4f9beef8 Jul 30 04:16:12.640993 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.640970 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2wmt\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641097 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641040 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641097 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641081 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641189 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641106 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641189 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641157 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641197 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641219 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641243 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641433 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641296 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert\") pod \"8d6b0599-e37d-4795-afed-f89ee2c43708\" (UID: \"8d6b0599-e37d-4795-afed-f89ee2c43708\") " Jul 30 04:16:12.641433 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641331 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:12.642056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641641 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.642056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641634 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:12.642056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.641982 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:12.642056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.642004 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:16:12.642538 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.642054 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:12.643788 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.643758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt" (OuterVolumeSpecName: "kube-api-access-w2wmt") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "kube-api-access-w2wmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:16:12.643947 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.643900 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:12.644153 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.644081 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:16:12.644265 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.644247 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d6b0599-e37d-4795-afed-f89ee2c43708" (UID: "8d6b0599-e37d-4795-afed-f89ee2c43708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742278 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6b0599-e37d-4795-afed-f89ee2c43708-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742307 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wmt\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-kube-api-access-w2wmt\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742316 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742325 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742333 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742340 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742348 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.742363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.742357 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6b0599-e37d-4795-afed-f89ee2c43708-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:12.935849 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.935810 2568 generic.go:356] "Generic (PLEG): container finished" podID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerID="32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad" exitCode=0 Jul 30 04:16:12.936035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.935872 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" Jul 30 04:16:12.936035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.935883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerDied","Data":"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad"} Jul 30 04:16:12.936035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.935939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv" event={"ID":"8d6b0599-e37d-4795-afed-f89ee2c43708","Type":"ContainerDied","Data":"a901fd62417fb425e8c853047fae7028fb3572ccb6d2738bb816bee7d1791f58"} Jul 30 04:16:12.936035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.935962 2568 scope.go:117] "RemoveContainer" containerID="32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad" Jul 30 04:16:12.937605 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.937584 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" event={"ID":"96d9720e-5a18-46b2-a23b-6212034fe22f","Type":"ContainerStarted","Data":"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0"} Jul 30 04:16:12.937730 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.937624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" event={"ID":"96d9720e-5a18-46b2-a23b-6212034fe22f","Type":"ContainerStarted","Data":"d0489a31fb3e3c3ded178e8953eada75f853f3242274af03db7791bc4f9beef8"} Jul 30 04:16:12.937730 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.937665 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:12.945867 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.945841 2568 scope.go:117] "RemoveContainer" containerID="edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c" Jul 30 04:16:12.954279 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.954260 2568 scope.go:117] "RemoveContainer" containerID="32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad" Jul 30 04:16:12.954832 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:16:12.954783 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad\": container with ID starting with 32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad not found: ID does not exist" containerID="32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad" Jul 30 04:16:12.954933 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.954869 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad"} err="failed to get container status \"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad\": rpc error: code = NotFound desc = could not find container \"32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad\": container with ID starting with 32ca7b9a0229a4861ac0ba0b6213f7e89949c0cbedc7ee24acdbd4495f1de0ad not found: ID does not exist" Jul 30 04:16:12.954933 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.954895 2568 scope.go:117] "RemoveContainer" containerID="edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c" Jul 30 04:16:12.955256 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:16:12.955186 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c\": container with ID starting with edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c not found: ID does not exist" containerID="edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c" Jul 30 04:16:12.955256 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.955231 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c"} err="failed to get container status \"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c\": rpc error: code = NotFound desc = could not find container \"edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c\": container with ID starting with edb042d8303a351d449bc484b287b7c594f6ef3ef591ec08aa01c6d1015eeb5c not found: ID does not exist" Jul 30 04:16:12.968994 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.968950 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" podStartSLOduration=0.968935419 podStartE2EDuration="968.935419ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:16:12.95560254 +0000 UTC m=+1853.001101740" watchObservedRunningTime="2026-07-30 04:16:12.968935419 +0000 UTC m=+1853.014434625" Jul 30 04:16:12.969542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.969526 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:16:12.973176 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:12.973152 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-974e458f-gw-openshift-default-658d5874d8-lrmlv"] Jul 30 04:16:14.569676 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:14.569628 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" path="/var/lib/kubelet/pods/8d6b0599-e37d-4795-afed-f89ee2c43708/volumes" Jul 30 04:16:23.947682 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:23.947643 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:24.972710 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:24.972117 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:60860->10.134.0.41:15021: read: connection reset by peer" start-of-body= Jul 30 04:16:24.972710 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:24.972223 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:60860->10.134.0.41:15021: read: connection reset by peer" Jul 30 04:16:26.012640 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.012605 2568 generic.go:356] "Generic (PLEG): container finished" podID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerID="aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251" exitCode=0 Jul 30 04:16:26.013050 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.012677 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerDied","Data":"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251"} Jul 30 04:16:26.013096 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.013062 2568 scope.go:117] "RemoveContainer" containerID="aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251" Jul 30 04:16:26.014062 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.014034 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:26.014162 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.014096 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:26.014162 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:26.014124 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:27.019441 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:27.019408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerStarted","Data":"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4"} Jul 30 04:16:27.341167 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:27.341069 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:27.341401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:27.341375 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 30 04:16:27.341457 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:27.341428 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 30 04:16:28.340553 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:28.340519 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 30 04:16:28.340949 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:28.340573 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 30 04:16:29.341195 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:29.341160 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 30 04:16:29.341661 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:29.341217 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 30 04:16:30.341074 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:30.341033 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 30 04:16:30.341267 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:30.341110 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 30 04:16:31.344508 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:31.344460 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:31.345007 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:31.344882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:31.345482 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:31.345453 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:36.125261 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.125218 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j"] Jul 30 04:16:36.125862 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.125843 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.125917 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.125868 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.125955 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.125916 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.125955 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.125925 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.126043 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.126026 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.126043 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.126040 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6b0599-e37d-4795-afed-f89ee2c43708" containerName="istio-proxy" Jul 30 04:16:36.128771 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.128752 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:36.131114 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.131092 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:16:36.131208 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.131121 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:16:36.137721 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.137690 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j"] Jul 30 04:16:36.284102 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.284062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j\" (UID: \"151d8be3-5a25-4b4b-9dc1-750cd3661953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:36.385847 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.385739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j\" (UID: \"151d8be3-5a25-4b4b-9dc1-750cd3661953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:36.386541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.386523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j\" (UID: \"151d8be3-5a25-4b4b-9dc1-750cd3661953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:36.448067 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.448033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:36.584059 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:36.584033 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j"] Jul 30 04:16:36.585909 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:16:36.585834 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151d8be3_5a25_4b4b_9dc1_750cd3661953.slice/crio-34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30 WatchSource:0}: Error finding container 34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30: Status 404 returned error can't find the container with id 34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30 Jul 30 04:16:37.070546 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:37.070423 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" event={"ID":"151d8be3-5a25-4b4b-9dc1-750cd3661953","Type":"ContainerStarted","Data":"f03d68c62e203ffeaf6d2f43436cab335751dedbaca491a469539ac41a06e8e0"} Jul 30 04:16:37.070546 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:37.070514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" event={"ID":"151d8be3-5a25-4b4b-9dc1-750cd3661953","Type":"ContainerStarted","Data":"34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30"} Jul 30 04:16:37.086186 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:37.086137 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" podStartSLOduration=1.086119799 podStartE2EDuration="1.086119799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:16:37.084817136 +0000 UTC m=+1877.130316323" watchObservedRunningTime="2026-07-30 04:16:37.086119799 +0000 UTC m=+1877.131618994" Jul 30 04:16:38.077547 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:38.077510 2568 generic.go:356] "Generic (PLEG): container finished" podID="151d8be3-5a25-4b4b-9dc1-750cd3661953" containerID="f03d68c62e203ffeaf6d2f43436cab335751dedbaca491a469539ac41a06e8e0" exitCode=0 Jul 30 04:16:38.077914 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:38.077606 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" event={"ID":"151d8be3-5a25-4b4b-9dc1-750cd3661953","Type":"ContainerDied","Data":"f03d68c62e203ffeaf6d2f43436cab335751dedbaca491a469539ac41a06e8e0"} Jul 30 04:16:39.214297 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:39.214269 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:39.313338 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:39.313303 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca\") pod \"151d8be3-5a25-4b4b-9dc1-750cd3661953\" (UID: \"151d8be3-5a25-4b4b-9dc1-750cd3661953\") " Jul 30 04:16:39.313748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:39.313728 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "151d8be3-5a25-4b4b-9dc1-750cd3661953" (UID: "151d8be3-5a25-4b4b-9dc1-750cd3661953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:16:39.414958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:39.414874 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151d8be3-5a25-4b4b-9dc1-750cd3661953-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:40.088638 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:40.088603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" event={"ID":"151d8be3-5a25-4b4b-9dc1-750cd3661953","Type":"ContainerDied","Data":"34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30"} Jul 30 04:16:40.088638 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:40.088621 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j" Jul 30 04:16:40.088857 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:40.088651 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d29fa16cc197f4cafb87c211eb24a5786165a86cfaf8c227da2f482bce5c30" Jul 30 04:16:41.986312 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:41.986280 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:41.987128 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:41.987074 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" podUID="96d9720e-5a18-46b2-a23b-6212034fe22f" containerName="payload-pre-processing" containerID="cri-o://6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0" gracePeriod=30 Jul 30 04:16:42.146270 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.146231 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j"] Jul 30 04:16:42.150786 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.150746 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b51d8f16-xf64j"] Jul 30 04:16:42.272124 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.272100 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:42.342673 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.342636 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5tzr\" (UniqueName: \"kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr\") pod \"96d9720e-5a18-46b2-a23b-6212034fe22f\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " Jul 30 04:16:42.342851 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.342697 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume\") pod \"96d9720e-5a18-46b2-a23b-6212034fe22f\" (UID: \"96d9720e-5a18-46b2-a23b-6212034fe22f\") " Jul 30 04:16:42.343121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.343093 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96d9720e-5a18-46b2-a23b-6212034fe22f" (UID: "96d9720e-5a18-46b2-a23b-6212034fe22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:16:42.344925 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.344901 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr" (OuterVolumeSpecName: "kube-api-access-q5tzr") pod "96d9720e-5a18-46b2-a23b-6212034fe22f" (UID: "96d9720e-5a18-46b2-a23b-6212034fe22f"). InnerVolumeSpecName "kube-api-access-q5tzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:16:42.443759 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.443723 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5tzr\" (UniqueName: \"kubernetes.io/projected/96d9720e-5a18-46b2-a23b-6212034fe22f-kube-api-access-q5tzr\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:42.443759 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.443751 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d9720e-5a18-46b2-a23b-6212034fe22f-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:42.568064 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:42.567979 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151d8be3-5a25-4b4b-9dc1-750cd3661953" path="/var/lib/kubelet/pods/151d8be3-5a25-4b4b-9dc1-750cd3661953/volumes" Jul 30 04:16:43.107183 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.107151 2568 generic.go:356] "Generic (PLEG): container finished" podID="96d9720e-5a18-46b2-a23b-6212034fe22f" containerID="6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0" exitCode=0 Jul 30 04:16:43.107765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.107196 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" Jul 30 04:16:43.107765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.107248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" event={"ID":"96d9720e-5a18-46b2-a23b-6212034fe22f","Type":"ContainerDied","Data":"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0"} Jul 30 04:16:43.107765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.107300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch" event={"ID":"96d9720e-5a18-46b2-a23b-6212034fe22f","Type":"ContainerDied","Data":"d0489a31fb3e3c3ded178e8953eada75f853f3242274af03db7791bc4f9beef8"} Jul 30 04:16:43.107765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.107324 2568 scope.go:117] "RemoveContainer" containerID="6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0" Jul 30 04:16:43.116096 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.116080 2568 scope.go:117] "RemoveContainer" containerID="6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0" Jul 30 04:16:43.116377 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:16:43.116358 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0\": container with ID starting with 6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0 not found: ID does not exist" containerID="6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0" Jul 30 04:16:43.116447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.116392 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0"} err="failed to get container status \"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0\": rpc error: code = NotFound desc = could not find container \"6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0\": container with ID starting with 6c6496a3d9c13a6446597170c09fdbdbf45774a69dcac7d6b0aa2d73b92a34c0 not found: ID does not exist" Jul 30 04:16:43.126996 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.126974 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:43.135039 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.135015 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b51d8f16-7d864fd5c6-5mxch"] Jul 30 04:16:43.358838 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.358757 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:43.359059 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:43.359027 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" containerID="cri-o://e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4" gracePeriod=30 Jul 30 04:16:44.159327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159297 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:16:44.159821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159778 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d9720e-5a18-46b2-a23b-6212034fe22f" containerName="payload-pre-processing" Jul 30 04:16:44.159821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159792 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d9720e-5a18-46b2-a23b-6212034fe22f" containerName="payload-pre-processing" Jul 30 04:16:44.159821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159817 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151d8be3-5a25-4b4b-9dc1-750cd3661953" containerName="revoke-keys" Jul 30 04:16:44.159821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159822 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="151d8be3-5a25-4b4b-9dc1-750cd3661953" containerName="revoke-keys" Jul 30 04:16:44.160032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159877 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="151d8be3-5a25-4b4b-9dc1-750cd3661953" containerName="revoke-keys" Jul 30 04:16:44.160032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.159890 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d9720e-5a18-46b2-a23b-6212034fe22f" containerName="payload-pre-processing" Jul 30 04:16:44.163039 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.163020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.165167 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.165146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-484ee705-openshift-default-dockercfg-2c72m\"" Jul 30 04:16:44.172342 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.172319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:16:44.263787 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.263738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.263967 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.263807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.263967 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.263890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.263961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.263990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn25n\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.264055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264267 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.264111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264267 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.264183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.264267 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.264210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365409 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365409 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn25n\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.365939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365868 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.366102 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.365934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.366274 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.366252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.366416 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.366310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.366605 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.366586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.368276 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.368250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.368565 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.368541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.373632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.373609 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn25n\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.373741 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.373664 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token\") pod \"e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.474930 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.474892 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:44.570581 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.570548 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d9720e-5a18-46b2-a23b-6212034fe22f" path="/var/lib/kubelet/pods/96d9720e-5a18-46b2-a23b-6212034fe22f/volumes" Jul 30 04:16:44.616821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.616797 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:16:44.618887 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:16:44.618862 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a737852_f70b_4594_9cfc_68dd29a277b3.slice/crio-439b436245a3029284126ee4d69f5e7b35edd0a81df295700e04c25afd16fb42 WatchSource:0}: Error finding container 439b436245a3029284126ee4d69f5e7b35edd0a81df295700e04c25afd16fb42: Status 404 returned error can't find the container with id 439b436245a3029284126ee4d69f5e7b35edd0a81df295700e04c25afd16fb42 Jul 30 04:16:44.621178 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.621139 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:44.621273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.621230 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:44.621316 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:44.621272 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:16:45.121510 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:45.121457 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerStarted","Data":"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c"} Jul 30 04:16:45.121690 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:45.121520 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerStarted","Data":"439b436245a3029284126ee4d69f5e7b35edd0a81df295700e04c25afd16fb42"} Jul 30 04:16:45.141693 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:45.141641 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podStartSLOduration=1.141626298 podStartE2EDuration="1.141626298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:16:45.140187598 +0000 UTC m=+1885.185686796" watchObservedRunningTime="2026-07-30 04:16:45.141626298 +0000 UTC m=+1885.187125493" Jul 30 04:16:45.475482 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:45.475441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:45.480515 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:45.480490 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:46.127412 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:46.127381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:46.128509 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:46.128492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:16:46.347457 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:46.347420 2568 patch_prober.go:28] interesting pod/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:16:46.347644 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:46.347495 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:16:48.640816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.640790 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:48.715992 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.715961 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g89th\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716015 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716090 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716140 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716160 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716202 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716195 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716242 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716272 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716290 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket\") pod \"34790bc1-27fc-465d-a56e-dcadc354ea81\" (UID: \"34790bc1-27fc-465d-a56e-dcadc354ea81\") " Jul 30 04:16:48.716524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716306 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:48.716748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716549 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:16:48.716748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716609 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:48.716748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716726 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.716748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716746 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34790bc1-27fc-465d-a56e-dcadc354ea81-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.717109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716758 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.717109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.716795 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:48.717494 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.717447 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data" (OuterVolumeSpecName: "istio-data") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:48.718537 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.718498 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:16:48.718646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.718617 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token" (OuterVolumeSpecName: "istio-token") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:16:48.718709 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.718644 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th" (OuterVolumeSpecName: "kube-api-access-g89th") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "kube-api-access-g89th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:16:48.718831 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.718814 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34790bc1-27fc-465d-a56e-dcadc354ea81" (UID: "34790bc1-27fc-465d-a56e-dcadc354ea81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:16:48.817956 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817910 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.817956 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817948 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.817956 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817959 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34790bc1-27fc-465d-a56e-dcadc354ea81-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.818243 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817970 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89th\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-kube-api-access-g89th\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.818243 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817980 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:48.818243 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:48.817988 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34790bc1-27fc-465d-a56e-dcadc354ea81-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:16:49.144986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.144949 2568 generic.go:356] "Generic (PLEG): container finished" podID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerID="e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4" exitCode=0 Jul 30 04:16:49.145170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.145001 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" Jul 30 04:16:49.145170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.145030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerDied","Data":"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4"} Jul 30 04:16:49.145170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.145087 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz" event={"ID":"34790bc1-27fc-465d-a56e-dcadc354ea81","Type":"ContainerDied","Data":"29a148e806b6f1ef58d9e9b7d342db7f295b09708da240a634afd8e807fe3907"} Jul 30 04:16:49.145170 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.145111 2568 scope.go:117] "RemoveContainer" containerID="e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4" Jul 30 04:16:49.154937 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.154919 2568 scope.go:117] "RemoveContainer" containerID="aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251" Jul 30 04:16:49.163278 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.163257 2568 scope.go:117] "RemoveContainer" containerID="e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4" Jul 30 04:16:49.167410 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.167285 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:49.168112 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:16:49.168080 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4\": container with ID starting with e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4 not found: ID does not exist" containerID="e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4" Jul 30 04:16:49.168283 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.168122 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4"} err="failed to get container status \"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4\": rpc error: code = NotFound desc = could not find container \"e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4\": container with ID starting with e9d01c5786ccb5e2352ee2db51ddc697fb0584255ee96f4017410656ee8aa1f4 not found: ID does not exist" Jul 30 04:16:49.168283 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.168148 2568 scope.go:117] "RemoveContainer" containerID="aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251" Jul 30 04:16:49.168702 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:16:49.168670 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251\": container with ID starting with aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251 not found: ID does not exist" containerID="aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251" Jul 30 04:16:49.168861 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.168704 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251"} err="failed to get container status \"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251\": rpc error: code = NotFound desc = could not find container \"aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251\": container with ID starting with aa886231af5c4916b975d5a041fac5b5b96d1ec1d35ff137c78e88cb26a2e251 not found: ID does not exist" Jul 30 04:16:49.171373 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:49.171353 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b51d8f16-openshift-default-5d85849886-jbnsz"] Jul 30 04:16:50.569167 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:16:50.569127 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" path="/var/lib/kubelet/pods/34790bc1-27fc-465d-a56e-dcadc354ea81/volumes" Jul 30 04:17:00.218511 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.218460 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerID="1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c" exitCode=0 Jul 30 04:17:00.218926 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.218538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerDied","Data":"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c"} Jul 30 04:17:00.218969 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.218955 2568 scope.go:117] "RemoveContainer" containerID="1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c" Jul 30 04:17:00.220035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.220010 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:17:00.220103 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.220076 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:17:00.220140 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:00.220111 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:17:01.127511 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:01.127445 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:01.225932 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:01.225893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerStarted","Data":"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1"} Jul 30 04:17:01.475315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:01.475273 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:01.476638 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:01.476618 2568 patch_prober.go:28] interesting pod/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 30 04:17:01.476705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:01.476669 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 30 04:17:02.475495 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:02.475429 2568 patch_prober.go:28] interesting pod/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 30 04:17:02.475889 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:02.475513 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 30 04:17:03.475844 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:03.475807 2568 patch_prober.go:28] interesting pod/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 30 04:17:03.476309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:03.475863 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 30 04:17:04.475718 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:04.475680 2568 patch_prober.go:28] interesting pod/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 30 04:17:04.475909 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:04.475744 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 30 04:17:05.479273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:05.479232 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:05.479763 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:05.479750 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:05.480485 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:05.480444 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:11.402015 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.401983 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b"] Jul 30 04:17:11.402443 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.402429 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:17:11.402524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.402445 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:17:11.402564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.402553 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:17:11.402598 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.402567 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:17:11.405073 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.405057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:11.407849 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.407557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:17:11.407849 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.407740 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:17:11.413853 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.413510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b"] Jul 30 04:17:11.446610 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.446572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-484ee705-ks94b\" (UID: \"da51f95e-6d78-4566-bfe9-ebb43ed74b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:11.547087 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.547047 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-484ee705-ks94b\" (UID: \"da51f95e-6d78-4566-bfe9-ebb43ed74b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:11.547834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.547811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-484ee705-ks94b\" (UID: \"da51f95e-6d78-4566-bfe9-ebb43ed74b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:11.720008 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.719974 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:11.854263 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:11.854235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b"] Jul 30 04:17:11.857180 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:17:11.857152 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda51f95e_6d78_4566_bfe9_ebb43ed74b93.slice/crio-28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b WatchSource:0}: Error finding container 28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b: Status 404 returned error can't find the container with id 28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b Jul 30 04:17:12.282692 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:12.282658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" event={"ID":"da51f95e-6d78-4566-bfe9-ebb43ed74b93","Type":"ContainerStarted","Data":"d58bd3f630a5645a5658e4ccf40d4c07e26bafbacdaf7c37f6e013809b6e514d"} Jul 30 04:17:12.282880 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:12.282711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" event={"ID":"da51f95e-6d78-4566-bfe9-ebb43ed74b93","Type":"ContainerStarted","Data":"28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b"} Jul 30 04:17:12.296051 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:12.295993 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" podStartSLOduration=1.295977703 podStartE2EDuration="1.295977703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:17:12.29516827 +0000 UTC m=+1912.340667462" watchObservedRunningTime="2026-07-30 04:17:12.295977703 +0000 UTC m=+1912.341476898" Jul 30 04:17:13.289924 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:13.289884 2568 generic.go:356] "Generic (PLEG): container finished" podID="da51f95e-6d78-4566-bfe9-ebb43ed74b93" containerID="d58bd3f630a5645a5658e4ccf40d4c07e26bafbacdaf7c37f6e013809b6e514d" exitCode=0 Jul 30 04:17:13.290318 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:13.289947 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" event={"ID":"da51f95e-6d78-4566-bfe9-ebb43ed74b93","Type":"ContainerDied","Data":"d58bd3f630a5645a5658e4ccf40d4c07e26bafbacdaf7c37f6e013809b6e514d"} Jul 30 04:17:14.433353 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:14.433330 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:14.469673 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:14.469640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca\") pod \"da51f95e-6d78-4566-bfe9-ebb43ed74b93\" (UID: \"da51f95e-6d78-4566-bfe9-ebb43ed74b93\") " Jul 30 04:17:14.470003 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:14.469979 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da51f95e-6d78-4566-bfe9-ebb43ed74b93" (UID: "da51f95e-6d78-4566-bfe9-ebb43ed74b93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:17:14.570675 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:14.570358 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da51f95e-6d78-4566-bfe9-ebb43ed74b93-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:15.304136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:15.304106 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" Jul 30 04:17:15.304136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:15.304125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b" event={"ID":"da51f95e-6d78-4566-bfe9-ebb43ed74b93","Type":"ContainerDied","Data":"28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b"} Jul 30 04:17:15.304368 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:15.304169 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ae66c7f2bb31a7113e904abd91b824005d4b959a5339081ed02bf0170ad94b" Jul 30 04:17:16.850946 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:16.850907 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b"] Jul 30 04:17:16.853705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:16.853680 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-484ee705-ks94b"] Jul 30 04:17:16.990478 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:16.990428 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:17:16.990760 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:16.990704 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" containerID="cri-o://bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1" gracePeriod=30 Jul 30 04:17:18.567901 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:18.567864 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da51f95e-6d78-4566-bfe9-ebb43ed74b93" path="/var/lib/kubelet/pods/da51f95e-6d78-4566-bfe9-ebb43ed74b93/volumes" Jul 30 04:17:20.482401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:20.482360 2568 patch_prober.go:28] interesting pod/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:17:20.482808 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:20.482420 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:17:22.252549 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.252480 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:22.343421 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343393 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343647 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343437 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343647 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343520 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343647 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343557 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343647 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343578 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343678 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343712 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.343870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343773 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:17:22.343870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343792 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn25n\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n\") pod \"3a737852-f70b-4594-9cfc-68dd29a277b3\" (UID: \"3a737852-f70b-4594-9cfc-68dd29a277b3\") " Jul 30 04:17:22.344125 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:17:22.344125 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.343877 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:17:22.344229 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.344156 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:17:22.344447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.344374 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.344447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.344396 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.344447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.344411 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.344829 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.344781 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:17:22.345109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.345083 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerID="bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1" exitCode=0 Jul 30 04:17:22.345206 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.345142 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" Jul 30 04:17:22.345206 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.345169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerDied","Data":"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1"} Jul 30 04:17:22.345379 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.345229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8" event={"ID":"3a737852-f70b-4594-9cfc-68dd29a277b3","Type":"ContainerDied","Data":"439b436245a3029284126ee4d69f5e7b35edd0a81df295700e04c25afd16fb42"} Jul 30 04:17:22.345379 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.345263 2568 scope.go:117] "RemoveContainer" containerID="bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1" Jul 30 04:17:22.346165 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.346135 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:17:22.346287 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.346204 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:17:22.346537 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.346518 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n" (OuterVolumeSpecName: "kube-api-access-dn25n") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "kube-api-access-dn25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:17:22.346598 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.346534 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a737852-f70b-4594-9cfc-68dd29a277b3" (UID: "3a737852-f70b-4594-9cfc-68dd29a277b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:17:22.365124 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.365105 2568 scope.go:117] "RemoveContainer" containerID="1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c" Jul 30 04:17:22.373498 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.373458 2568 scope.go:117] "RemoveContainer" containerID="bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1" Jul 30 04:17:22.373802 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:17:22.373781 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1\": container with ID starting with bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1 not found: ID does not exist" containerID="bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1" Jul 30 04:17:22.373856 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.373813 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1"} err="failed to get container status \"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1\": rpc error: code = NotFound desc = could not find container \"bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1\": container with ID starting with bdb0cf13c1e594b5173121484cbb2df96e624b0fb11317e46063fa1085e175f1 not found: ID does not exist" Jul 30 04:17:22.373856 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.373830 2568 scope.go:117] "RemoveContainer" containerID="1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c" Jul 30 04:17:22.374115 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:17:22.374085 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c\": container with ID starting with 1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c not found: ID does not exist" containerID="1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c" Jul 30 04:17:22.374180 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.374124 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c"} err="failed to get container status \"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c\": rpc error: code = NotFound desc = could not find container \"1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c\": container with ID starting with 1b1f38cae2604acc7913d1657a414d5a7e85ec158fec09af14f86a424709158c not found: ID does not exist" Jul 30 04:17:22.445913 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445878 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.445913 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445908 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.445913 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445919 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.446151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445929 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a737852-f70b-4594-9cfc-68dd29a277b3-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.446151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445937 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn25n\" (UniqueName: \"kubernetes.io/projected/3a737852-f70b-4594-9cfc-68dd29a277b3-kube-api-access-dn25n\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.446151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.445947 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a737852-f70b-4594-9cfc-68dd29a277b3-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:17:22.661088 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.661053 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:17:22.664893 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:22.664863 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-484ee705-openshift-default-65ccc68dd4-vb9p8"] Jul 30 04:17:24.568847 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:17:24.568816 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" path="/var/lib/kubelet/pods/3a737852-f70b-4594-9cfc-68dd29a277b3/volumes" Jul 30 04:19:24.856340 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856301 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856725 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856737 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856748 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856753 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="34790bc1-27fc-465d-a56e-dcadc354ea81" containerName="istio-proxy" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856769 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da51f95e-6d78-4566-bfe9-ebb43ed74b93" containerName="revoke-keys" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856775 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da51f95e-6d78-4566-bfe9-ebb43ed74b93" containerName="revoke-keys" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856827 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:24.856840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.856835 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="da51f95e-6d78-4566-bfe9-ebb43ed74b93" containerName="revoke-keys" Jul 30 04:19:24.859016 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.858999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.864861 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.864840 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9a3256f1-openshift-default-dockercfg-zhjp4\"" Jul 30 04:19:24.880006 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.879934 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:19:24.975483 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975656 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975656 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975656 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975764 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975764 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975705 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975830 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975830 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rcz\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:24.975897 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:24.975834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076513 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076513 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rcz\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076773 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076773 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076885 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076885 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.076885 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.076978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.077011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077219 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.077180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077291 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.077266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.077299 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.077725 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.077703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.079106 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.079086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.079313 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.079295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.084057 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.084026 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rcz\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.084480 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.084443 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token\") pod \"e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.169293 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.169261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:25.313439 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.313273 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:19:25.316080 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:19:25.316054 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb1aae7_30fe_4ab6_b3be_1588aad0c673.slice/crio-210e2cfa6a93f5c5718e574a7b34e325bab3e26569ff043a93106a472bd02a07 WatchSource:0}: Error finding container 210e2cfa6a93f5c5718e574a7b34e325bab3e26569ff043a93106a472bd02a07: Status 404 returned error can't find the container with id 210e2cfa6a93f5c5718e574a7b34e325bab3e26569ff043a93106a472bd02a07 Jul 30 04:19:25.318738 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.318702 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:25.318852 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.318784 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:25.318852 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.318831 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:25.989218 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.989183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerStarted","Data":"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54"} Jul 30 04:19:25.989748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:25.989240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerStarted","Data":"210e2cfa6a93f5c5718e574a7b34e325bab3e26569ff043a93106a472bd02a07"} Jul 30 04:19:26.009504 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:26.009425 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podStartSLOduration=2.009402949 podStartE2EDuration="2.009402949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:26.008336508 +0000 UTC m=+2046.053835703" watchObservedRunningTime="2026-07-30 04:19:26.009402949 +0000 UTC m=+2046.054902147" Jul 30 04:19:26.169836 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:26.169795 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:26.174438 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:26.174413 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:26.995506 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:26.995456 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:26.996366 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:26.996345 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:29.748035 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.747993 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:19:29.748436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.748385 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:29.748436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.748397 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:29.748551 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.748517 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a737852-f70b-4594-9cfc-68dd29a277b3" containerName="istio-proxy" Jul 30 04:19:29.750728 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.750708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:29.753280 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.753258 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9a3256f1-dockercfg-l4754\"" Jul 30 04:19:29.753371 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.753304 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9a3256f1\"" Jul 30 04:19:29.762882 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.762854 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:19:29.768264 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.768238 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:19:29.771067 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.771044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:29.782054 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.782034 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:19:29.936897 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.936856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:29.936897 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.936898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:29.937105 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.937035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7lm\" (UniqueName: \"kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:29.937105 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:29.937075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4j6\" (UniqueName: \"kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.038728 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.038625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.038728 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.038669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:30.038978 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.038740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7lm\" (UniqueName: \"kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:30.038978 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.038770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4j6\" (UniqueName: \"kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.039382 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.039362 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.039438 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.039385 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:30.050051 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.049997 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4j6\" (UniqueName: \"kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6\") pod \"payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.057975 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.057917 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7lm\" (UniqueName: \"kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm\") pod \"payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:30.063374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.063351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:30.082974 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.082948 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:30.218405 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.218366 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:19:30.223020 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:19:30.222982 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5eb8a7_4678_4491_8392_5abacff5ef4b.slice/crio-708f6f13cf54d5e3a7d20b8edf5a721a78ec432a74fec4eeb2fd71fde653d2db WatchSource:0}: Error finding container 708f6f13cf54d5e3a7d20b8edf5a721a78ec432a74fec4eeb2fd71fde653d2db: Status 404 returned error can't find the container with id 708f6f13cf54d5e3a7d20b8edf5a721a78ec432a74fec4eeb2fd71fde653d2db Jul 30 04:19:30.242557 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:30.242526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:19:30.245135 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:19:30.245110 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c198bb_9398_4be5_aacb_533d3596c611.slice/crio-26e52452377c2d7c2b2ee780b180d3d0c245e33334c09f543927ff975c327617 WatchSource:0}: Error finding container 26e52452377c2d7c2b2ee780b180d3d0c245e33334c09f543927ff975c327617: Status 404 returned error can't find the container with id 26e52452377c2d7c2b2ee780b180d3d0c245e33334c09f543927ff975c327617 Jul 30 04:19:31.017608 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.017575 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" event={"ID":"0c5eb8a7-4678-4491-8392-5abacff5ef4b","Type":"ContainerStarted","Data":"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f"} Jul 30 04:19:31.018106 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.017638 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" event={"ID":"0c5eb8a7-4678-4491-8392-5abacff5ef4b","Type":"ContainerStarted","Data":"708f6f13cf54d5e3a7d20b8edf5a721a78ec432a74fec4eeb2fd71fde653d2db"} Jul 30 04:19:31.018106 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.017683 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:31.019171 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.019145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" event={"ID":"64c198bb-9398-4be5-aacb-533d3596c611","Type":"ContainerStarted","Data":"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86"} Jul 30 04:19:31.019297 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.019195 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:31.019297 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.019215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" event={"ID":"64c198bb-9398-4be5-aacb-533d3596c611","Type":"ContainerStarted","Data":"26e52452377c2d7c2b2ee780b180d3d0c245e33334c09f543927ff975c327617"} Jul 30 04:19:31.034516 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.034455 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" podStartSLOduration=2.034440723 podStartE2EDuration="2.034440723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:31.033310582 +0000 UTC m=+2051.078809778" watchObservedRunningTime="2026-07-30 04:19:31.034440723 +0000 UTC m=+2051.079939918" Jul 30 04:19:31.051524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:31.051417 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" podStartSLOduration=2.051399514 podStartE2EDuration="2.051399514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:31.050827399 +0000 UTC m=+2051.096326598" watchObservedRunningTime="2026-07-30 04:19:31.051399514 +0000 UTC m=+2051.096898709" Jul 30 04:19:41.997440 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:41.997392 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 30 04:19:41.997996 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:41.997458 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 30 04:19:42.026218 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.026180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:19:42.026405 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.026270 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:19:42.091539 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.091498 2568 generic.go:356] "Generic (PLEG): container finished" podID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerID="3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54" exitCode=0 Jul 30 04:19:42.091539 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.091538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerDied","Data":"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54"} Jul 30 04:19:42.091958 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.091943 2568 scope.go:117] "RemoveContainer" containerID="3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54" Jul 30 04:19:42.093097 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.093066 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:42.093179 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.093153 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:42.093227 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:42.093206 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:19:43.099778 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:43.099739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerStarted","Data":"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370"} Jul 30 04:19:43.169560 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:43.169515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:43.169816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:43.169787 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 30 04:19:43.169896 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:43.169864 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 30 04:19:44.170536 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:44.170460 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 30 04:19:44.171060 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:44.170561 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 30 04:19:45.169756 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:45.169715 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 30 04:19:45.170063 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:45.170031 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 30 04:19:46.170221 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:46.170174 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 30 04:19:46.170762 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:46.170244 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 30 04:19:47.174315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:47.174279 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:47.174740 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:47.174638 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:47.175252 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:47.175236 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:19:59.997665 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:19:59.997620 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n"] Jul 30 04:20:00.001676 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.001643 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:00.003898 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.003874 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:20:00.004036 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.003973 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:20:00.010282 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.010226 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n"] Jul 30 04:20:00.154177 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.154137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n\" (UID: \"0bd965da-82a6-463b-bb33-a7bf4e2dbd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:00.255810 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.255713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n\" (UID: \"0bd965da-82a6-463b-bb33-a7bf4e2dbd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:00.256362 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.256342 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n\" (UID: \"0bd965da-82a6-463b-bb33-a7bf4e2dbd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:00.313945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.313903 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:00.457452 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:00.457423 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n"] Jul 30 04:20:00.459972 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:20:00.459938 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd965da_82a6_463b_bb33_a7bf4e2dbd40.slice/crio-9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b WatchSource:0}: Error finding container 9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b: Status 404 returned error can't find the container with id 9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b Jul 30 04:20:01.197273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:01.197237 2568 generic.go:356] "Generic (PLEG): container finished" podID="0bd965da-82a6-463b-bb33-a7bf4e2dbd40" containerID="65ef3766a08eb57a663fdfbf9200411aedc74087cd25bc6e8feaab89886d527b" exitCode=0 Jul 30 04:20:01.197687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:01.197326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" event={"ID":"0bd965da-82a6-463b-bb33-a7bf4e2dbd40","Type":"ContainerDied","Data":"65ef3766a08eb57a663fdfbf9200411aedc74087cd25bc6e8feaab89886d527b"} Jul 30 04:20:01.197687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:01.197385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" event={"ID":"0bd965da-82a6-463b-bb33-a7bf4e2dbd40","Type":"ContainerStarted","Data":"9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b"} Jul 30 04:20:02.354149 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:02.354119 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:02.479915 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:02.479814 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca\") pod \"0bd965da-82a6-463b-bb33-a7bf4e2dbd40\" (UID: \"0bd965da-82a6-463b-bb33-a7bf4e2dbd40\") " Jul 30 04:20:02.480204 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:02.480175 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bd965da-82a6-463b-bb33-a7bf4e2dbd40" (UID: "0bd965da-82a6-463b-bb33-a7bf4e2dbd40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:20:02.581133 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:02.581098 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd965da-82a6-463b-bb33-a7bf4e2dbd40-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:03.209608 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:03.209575 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" Jul 30 04:20:03.209608 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:03.209596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n" event={"ID":"0bd965da-82a6-463b-bb33-a7bf4e2dbd40","Type":"ContainerDied","Data":"9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b"} Jul 30 04:20:03.209819 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:03.209669 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9696c225695dd46f1b4f37075e99b878b70c2420fcabec13091c5e47efe4933b" Jul 30 04:20:10.555404 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.555373 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:20:10.555835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.555618 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" podUID="64c198bb-9398-4be5-aacb-533d3596c611" containerName="payload-processing" containerID="cri-o://616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86" gracePeriod=30 Jul 30 04:20:10.572016 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.571574 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:20:10.572016 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.571807 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" podUID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" containerName="payload-pre-processing" containerID="cri-o://180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f" gracePeriod=30 Jul 30 04:20:10.699798 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.699200 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n"] Jul 30 04:20:10.702704 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.702670 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a3256f1-5h72n"] Jul 30 04:20:10.870333 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.870304 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:20:10.873196 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.873176 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:20:10.971773 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.971742 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume\") pod \"64c198bb-9398-4be5-aacb-533d3596c611\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " Jul 30 04:20:10.971989 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.971856 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume\") pod \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " Jul 30 04:20:10.971989 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.971887 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7lm\" (UniqueName: \"kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm\") pod \"64c198bb-9398-4be5-aacb-533d3596c611\" (UID: \"64c198bb-9398-4be5-aacb-533d3596c611\") " Jul 30 04:20:10.971989 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.971938 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l4j6\" (UniqueName: \"kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6\") pod \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\" (UID: \"0c5eb8a7-4678-4491-8392-5abacff5ef4b\") " Jul 30 04:20:10.972151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.972131 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64c198bb-9398-4be5-aacb-533d3596c611" (UID: "64c198bb-9398-4be5-aacb-533d3596c611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:20:10.972260 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.972233 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c5eb8a7-4678-4491-8392-5abacff5ef4b" (UID: "0c5eb8a7-4678-4491-8392-5abacff5ef4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:20:10.972358 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.972344 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c198bb-9398-4be5-aacb-533d3596c611-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:10.972396 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.972362 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5eb8a7-4678-4491-8392-5abacff5ef4b-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:10.974185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.974163 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm" (OuterVolumeSpecName: "kube-api-access-6r7lm") pod "64c198bb-9398-4be5-aacb-533d3596c611" (UID: "64c198bb-9398-4be5-aacb-533d3596c611"). InnerVolumeSpecName "kube-api-access-6r7lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:20:10.974256 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:10.974177 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6" (OuterVolumeSpecName: "kube-api-access-9l4j6") pod "0c5eb8a7-4678-4491-8392-5abacff5ef4b" (UID: "0c5eb8a7-4678-4491-8392-5abacff5ef4b"). InnerVolumeSpecName "kube-api-access-9l4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:20:11.073362 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.073330 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7lm\" (UniqueName: \"kubernetes.io/projected/64c198bb-9398-4be5-aacb-533d3596c611-kube-api-access-6r7lm\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:11.073362 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.073355 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4j6\" (UniqueName: \"kubernetes.io/projected/0c5eb8a7-4678-4491-8392-5abacff5ef4b-kube-api-access-9l4j6\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:11.254385 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.254345 2568 generic.go:356] "Generic (PLEG): container finished" podID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" containerID="180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f" exitCode=0 Jul 30 04:20:11.254618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.254392 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" Jul 30 04:20:11.254618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.254438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" event={"ID":"0c5eb8a7-4678-4491-8392-5abacff5ef4b","Type":"ContainerDied","Data":"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f"} Jul 30 04:20:11.254618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.254531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg" event={"ID":"0c5eb8a7-4678-4491-8392-5abacff5ef4b","Type":"ContainerDied","Data":"708f6f13cf54d5e3a7d20b8edf5a721a78ec432a74fec4eeb2fd71fde653d2db"} Jul 30 04:20:11.254618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.254565 2568 scope.go:117] "RemoveContainer" containerID="180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f" Jul 30 04:20:11.255967 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.255945 2568 generic.go:356] "Generic (PLEG): container finished" podID="64c198bb-9398-4be5-aacb-533d3596c611" containerID="616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86" exitCode=0 Jul 30 04:20:11.256083 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.255971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" event={"ID":"64c198bb-9398-4be5-aacb-533d3596c611","Type":"ContainerDied","Data":"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86"} Jul 30 04:20:11.256083 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.255996 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" Jul 30 04:20:11.256083 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.256023 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z" event={"ID":"64c198bb-9398-4be5-aacb-533d3596c611","Type":"ContainerDied","Data":"26e52452377c2d7c2b2ee780b180d3d0c245e33334c09f543927ff975c327617"} Jul 30 04:20:11.268089 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.268023 2568 scope.go:117] "RemoveContainer" containerID="180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f" Jul 30 04:20:11.268487 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:20:11.268439 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f\": container with ID starting with 180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f not found: ID does not exist" containerID="180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f" Jul 30 04:20:11.268561 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.268502 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f"} err="failed to get container status \"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f\": rpc error: code = NotFound desc = could not find container \"180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f\": container with ID starting with 180bfe3b2f5bffe483f5554cdef5d5747780a1ef9aee8797ed819cf1e3b97a9f not found: ID does not exist" Jul 30 04:20:11.268561 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.268528 2568 scope.go:117] "RemoveContainer" containerID="616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86" Jul 30 04:20:11.277199 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.277180 2568 scope.go:117] "RemoveContainer" containerID="616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86" Jul 30 04:20:11.277445 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:20:11.277427 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86\": container with ID starting with 616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86 not found: ID does not exist" containerID="616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86" Jul 30 04:20:11.277524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.277454 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86"} err="failed to get container status \"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86\": rpc error: code = NotFound desc = could not find container \"616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86\": container with ID starting with 616f07b1c7327b6c486f899a76f2e3dc570477c9fb91ae33ba38edf0bac96d86 not found: ID does not exist" Jul 30 04:20:11.277722 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.277703 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:20:11.283511 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.283491 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a3256f1-7f6b74796-t5csg"] Jul 30 04:20:11.294086 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.294056 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:20:11.296118 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:11.296095 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a3256f1-c8fbdcc74-jvb8z"] Jul 30 04:20:12.567247 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:12.567204 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd965da-82a6-463b-bb33-a7bf4e2dbd40" path="/var/lib/kubelet/pods/0bd965da-82a6-463b-bb33-a7bf4e2dbd40/volumes" Jul 30 04:20:12.567802 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:12.567780 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" path="/var/lib/kubelet/pods/0c5eb8a7-4678-4491-8392-5abacff5ef4b/volumes" Jul 30 04:20:12.568176 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:12.568156 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c198bb-9398-4be5-aacb-533d3596c611" path="/var/lib/kubelet/pods/64c198bb-9398-4be5-aacb-533d3596c611/volumes" Jul 30 04:20:16.941522 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:16.941483 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:20:16.941988 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:16.941724 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" containerID="cri-o://b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370" gracePeriod=30 Jul 30 04:20:17.177433 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:17.177393 2568 patch_prober.go:28] interesting pod/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:20:17.177636 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:17.177458 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:20:22.193065 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.193038 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:20:22.288168 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288062 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288168 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288173 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288226 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288290 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288331 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288368 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25rcz\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288697 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288416 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288697 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288528 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket\") pod \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\" (UID: \"bcb1aae7-30fe-4ab6-b3be-1588aad0c673\") " Jul 30 04:20:22.288697 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288594 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:20:22.288697 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288606 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:20:22.288906 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288692 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:20:22.289002 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.288958 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.289135 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.289117 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.289188 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.289139 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.289188 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.289151 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:20:22.289281 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.289218 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:20:22.290654 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.290626 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:20:22.290771 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.290658 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:20:22.291163 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.291147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz" (OuterVolumeSpecName: "kube-api-access-25rcz") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "kube-api-access-25rcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:20:22.291163 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.291152 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcb1aae7-30fe-4ab6-b3be-1588aad0c673" (UID: "bcb1aae7-30fe-4ab6-b3be-1588aad0c673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:20:22.316479 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.316435 2568 generic.go:356] "Generic (PLEG): container finished" podID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerID="b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370" exitCode=0 Jul 30 04:20:22.316600 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.316496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerDied","Data":"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370"} Jul 30 04:20:22.316600 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.316508 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" Jul 30 04:20:22.316600 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.316545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2" event={"ID":"bcb1aae7-30fe-4ab6-b3be-1588aad0c673","Type":"ContainerDied","Data":"210e2cfa6a93f5c5718e574a7b34e325bab3e26569ff043a93106a472bd02a07"} Jul 30 04:20:22.316600 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.316567 2568 scope.go:117] "RemoveContainer" containerID="b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370" Jul 30 04:20:22.325907 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.325890 2568 scope.go:117] "RemoveContainer" containerID="3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54" Jul 30 04:20:22.334562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.334542 2568 scope.go:117] "RemoveContainer" containerID="b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370" Jul 30 04:20:22.335423 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:20:22.335398 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370\": container with ID starting with b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370 not found: ID does not exist" containerID="b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370" Jul 30 04:20:22.335552 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.335430 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370"} err="failed to get container status \"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370\": rpc error: code = NotFound desc = could not find container \"b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370\": container with ID starting with b3497d649c29df01f5b75d9aed7d7c78b2a44009f4b8f87a1d75e224a39a8370 not found: ID does not exist" Jul 30 04:20:22.335552 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.335448 2568 scope.go:117] "RemoveContainer" containerID="3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54" Jul 30 04:20:22.335773 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:20:22.335755 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54\": container with ID starting with 3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54 not found: ID does not exist" containerID="3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54" Jul 30 04:20:22.335840 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.335779 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54"} err="failed to get container status \"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54\": rpc error: code = NotFound desc = could not find container \"3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54\": container with ID starting with 3c444420b06bc71ea3872fda94a2eea41c8f1c74e5b006d20fb416851ecaac54 not found: ID does not exist" Jul 30 04:20:22.337254 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.337231 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:20:22.341778 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.341753 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9a3256f1-openshift-default-8487db7448-wzbw2"] Jul 30 04:20:22.389750 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389727 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.389750 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389750 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.389911 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389760 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.389911 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389768 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.389911 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389777 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rcz\" (UniqueName: \"kubernetes.io/projected/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-kube-api-access-25rcz\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.389911 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.389786 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb1aae7-30fe-4ab6-b3be-1588aad0c673-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:20:22.568388 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:20:22.568309 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" path="/var/lib/kubelet/pods/bcb1aae7-30fe-4ab6-b3be-1588aad0c673/volumes" Jul 30 04:22:20.676188 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:20.676150 2568 scope.go:117] "RemoveContainer" containerID="9ddd118f8703724872ff777715679b290e83ab5776813891f131fc8bab0744e3" Jul 30 04:22:20.685047 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:20.685027 2568 scope.go:117] "RemoveContainer" containerID="5beb9325058338e98c15e607018452527bf7cb4cedb493bdc8ae84083aae844c" Jul 30 04:22:43.135315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135218 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:22:43.135903 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135883 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" containerName="payload-pre-processing" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135908 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" containerName="payload-pre-processing" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135925 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135934 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135953 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd965da-82a6-463b-bb33-a7bf4e2dbd40" containerName="revoke-keys" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135961 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd965da-82a6-463b-bb33-a7bf4e2dbd40" containerName="revoke-keys" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135985 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c198bb-9398-4be5-aacb-533d3596c611" containerName="payload-processing" Jul 30 04:22:43.135991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.135993 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c198bb-9398-4be5-aacb-533d3596c611" containerName="payload-processing" Jul 30 04:22:43.136327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.136119 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:43.136327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.136134 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5eb8a7-4678-4491-8392-5abacff5ef4b" containerName="payload-pre-processing" Jul 30 04:22:43.136327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.136147 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c198bb-9398-4be5-aacb-533d3596c611" containerName="payload-processing" Jul 30 04:22:43.136327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.136157 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd965da-82a6-463b-bb33-a7bf4e2dbd40" containerName="revoke-keys" Jul 30 04:22:43.139403 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.139384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.141719 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.141698 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bd1da342-openshift-default-dockercfg-5qwqx\"" Jul 30 04:22:43.150541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.150509 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:22:43.274703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.274854 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.274854 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.274854 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.275033 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.275033 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.274964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpwks\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.275143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.275049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.275143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.275092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.275143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.275122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376350 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376602 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376886 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376886 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.376886 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376686 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpwks\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.377032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.377032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.376943 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.377187 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.377163 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.377272 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.377252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.377325 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.377286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.379103 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.379080 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.379352 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.379331 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.398237 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.398173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpwks\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.398353 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.398252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token\") pod \"e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.450730 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.450695 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:43.590048 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.590013 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:22:43.593093 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:22:43.593067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c25e8c_a39e_44b2_9969_353109c2f9d1.slice/crio-2f076054e2617af8630420ea409371dc8fde579d71a1b9883b55cea3d7adc080 WatchSource:0}: Error finding container 2f076054e2617af8630420ea409371dc8fde579d71a1b9883b55cea3d7adc080: Status 404 returned error can't find the container with id 2f076054e2617af8630420ea409371dc8fde579d71a1b9883b55cea3d7adc080 Jul 30 04:22:43.595216 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.595186 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:22:43.595289 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.595248 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:22:43.595289 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:43.595281 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:22:44.052303 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:44.052271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerStarted","Data":"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e"} Jul 30 04:22:44.052495 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:44.052329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerStarted","Data":"2f076054e2617af8630420ea409371dc8fde579d71a1b9883b55cea3d7adc080"} Jul 30 04:22:44.071201 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:44.071149 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podStartSLOduration=1.071130318 podStartE2EDuration="1.071130318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:44.070293503 +0000 UTC m=+2244.115792694" watchObservedRunningTime="2026-07-30 04:22:44.071130318 +0000 UTC m=+2244.116629512" Jul 30 04:22:44.451498 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:44.451432 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:44.456988 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:44.456964 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:45.058161 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:45.058126 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:45.059264 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:45.059243 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:22:47.808242 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.808210 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:22:47.808646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.808628 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:47.808646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.808640 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:47.808727 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.808714 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb1aae7-30fe-4ab6-b3be-1588aad0c673" containerName="istio-proxy" Jul 30 04:22:47.811901 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.811882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:47.813841 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.813824 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bd1da342-dockercfg-zx6rm\"" Jul 30 04:22:47.813952 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.813933 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bd1da342\"" Jul 30 04:22:47.822168 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.822145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:22:47.925017 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.924971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:47.925190 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:47.925095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg8c\" (UniqueName: \"kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.026065 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.026021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg8c\" (UniqueName: \"kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.026266 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.026147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.026902 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.026883 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.033571 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.033546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg8c\" (UniqueName: \"kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c\") pod \"payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.123903 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.123818 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:48.264647 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:48.264619 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:22:48.266318 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:22:48.266291 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a541c_b238_403e_a12f_a76cd79404cc.slice/crio-ac8cc558b357409d9c8de5025c90dcdbb47b3f01fb9c3908b614579f72881890 WatchSource:0}: Error finding container ac8cc558b357409d9c8de5025c90dcdbb47b3f01fb9c3908b614579f72881890: Status 404 returned error can't find the container with id ac8cc558b357409d9c8de5025c90dcdbb47b3f01fb9c3908b614579f72881890 Jul 30 04:22:49.081965 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:49.081926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" event={"ID":"637a541c-b238-403e-a12f-a76cd79404cc","Type":"ContainerStarted","Data":"c465685b9a20f7bf36b33e89ed8cebda655f7dccc41f277c2b62b427f7ce125c"} Jul 30 04:22:49.082524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:49.081991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" event={"ID":"637a541c-b238-403e-a12f-a76cd79404cc","Type":"ContainerStarted","Data":"ac8cc558b357409d9c8de5025c90dcdbb47b3f01fb9c3908b614579f72881890"} Jul 30 04:22:49.082524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:49.082034 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:22:49.119574 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:22:49.119516 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" podStartSLOduration=2.119498284 podStartE2EDuration="2.119498284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:49.118124188 +0000 UTC m=+2249.163623408" watchObservedRunningTime="2026-07-30 04:22:49.119498284 +0000 UTC m=+2249.164997480" Jul 30 04:23:00.089724 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:00.089626 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:23:01.099660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:01.099624 2568 patch_prober.go:28] interesting pod/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 04:23:01.100143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:01.099693 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 04:23:03.156734 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.156649 2568 generic.go:356] "Generic (PLEG): container finished" podID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerID="de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e" exitCode=0 Jul 30 04:23:03.156734 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.156706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerDied","Data":"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e"} Jul 30 04:23:03.157165 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.157115 2568 scope.go:117] "RemoveContainer" containerID="de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e" Jul 30 04:23:03.158178 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.158144 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:23:03.158268 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.158216 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:23:03.158268 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:03.158247 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:23:04.167553 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:04.167503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerStarted","Data":"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3"} Jul 30 04:23:04.451191 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:04.451076 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:23:04.451422 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:04.451391 2568 patch_prober.go:28] interesting pod/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 30 04:23:04.451657 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:04.451625 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 30 04:23:05.452316 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:05.452272 2568 patch_prober.go:28] interesting pod/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 30 04:23:05.453130 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:05.452996 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 30 04:23:06.451992 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:06.451952 2568 patch_prober.go:28] interesting pod/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 30 04:23:06.452307 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:06.452278 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 30 04:23:07.451256 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:07.451210 2568 patch_prober.go:28] interesting pod/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 30 04:23:07.451801 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:07.451301 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 30 04:23:08.455015 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:08.454981 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:23:08.455452 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:08.455268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:23:08.456131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:08.456115 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:23:20.712069 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:20.712042 2568 scope.go:117] "RemoveContainer" containerID="f03d68c62e203ffeaf6d2f43436cab335751dedbaca491a469539ac41a06e8e0" Jul 30 04:23:20.721413 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:20.721391 2568 scope.go:117] "RemoveContainer" containerID="d58bd3f630a5645a5658e4ccf40d4c07e26bafbacdaf7c37f6e013809b6e514d" Jul 30 04:23:55.692182 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.692136 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8"] Jul 30 04:23:55.699902 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.699871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:55.703040 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.702857 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:23:55.703190 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.703074 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:23:55.703572 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.703519 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8"] Jul 30 04:23:55.801278 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.801243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd1da342-xszk8\" (UID: \"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:55.902765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.902734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd1da342-xszk8\" (UID: \"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:55.903359 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:55.903343 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd1da342-xszk8\" (UID: \"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:56.015138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:56.015053 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:56.146499 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:56.146447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8"] Jul 30 04:23:56.150272 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:23:56.150243 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca42643a_e5c6_4fcd_956e_9ec6b8e93f0a.slice/crio-98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422 WatchSource:0}: Error finding container 98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422: Status 404 returned error can't find the container with id 98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422 Jul 30 04:23:56.444689 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:56.444654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" event={"ID":"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a","Type":"ContainerStarted","Data":"097e6b88f3ed86c3b0f2b4e833493de7c514b9a0228b26fe885a9839e76d6c59"} Jul 30 04:23:56.444884 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:56.444703 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" event={"ID":"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a","Type":"ContainerStarted","Data":"98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422"} Jul 30 04:23:56.459542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:56.459452 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" podStartSLOduration=1.459430282 podStartE2EDuration="1.459430282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:23:56.45853663 +0000 UTC m=+2316.504035827" watchObservedRunningTime="2026-07-30 04:23:56.459430282 +0000 UTC m=+2316.504929479" Jul 30 04:23:57.451966 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:57.451930 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" containerID="097e6b88f3ed86c3b0f2b4e833493de7c514b9a0228b26fe885a9839e76d6c59" exitCode=0 Jul 30 04:23:57.452429 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:57.452022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" event={"ID":"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a","Type":"ContainerDied","Data":"097e6b88f3ed86c3b0f2b4e833493de7c514b9a0228b26fe885a9839e76d6c59"} Jul 30 04:23:58.593737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:58.593713 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:58.730512 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:58.730411 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca\") pod \"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a\" (UID: \"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a\") " Jul 30 04:23:58.730991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:58.730844 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" (UID: "ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:23:58.730991 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:58.730964 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:23:59.463892 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:59.463866 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" Jul 30 04:23:59.464067 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:59.463899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8" event={"ID":"ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a","Type":"ContainerDied","Data":"98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422"} Jul 30 04:23:59.464067 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:23:59.463941 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98cfbd99041e25ca528d5dcd11e8be0002def89056a710e54d3f66f718e41422" Jul 30 04:24:11.308886 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.308799 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:24:11.309299 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.309086 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" podUID="637a541c-b238-403e-a12f-a76cd79404cc" containerName="payload-pre-processing" containerID="cri-o://c465685b9a20f7bf36b33e89ed8cebda655f7dccc41f277c2b62b427f7ce125c" gracePeriod=30 Jul 30 04:24:11.436808 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.436777 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8"] Jul 30 04:24:11.439903 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.439877 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd1da342-xszk8"] Jul 30 04:24:11.532063 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.532027 2568 generic.go:356] "Generic (PLEG): container finished" podID="637a541c-b238-403e-a12f-a76cd79404cc" containerID="c465685b9a20f7bf36b33e89ed8cebda655f7dccc41f277c2b62b427f7ce125c" exitCode=0 Jul 30 04:24:11.532227 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.532094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" event={"ID":"637a541c-b238-403e-a12f-a76cd79404cc","Type":"ContainerDied","Data":"c465685b9a20f7bf36b33e89ed8cebda655f7dccc41f277c2b62b427f7ce125c"} Jul 30 04:24:11.570852 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.570784 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:24:11.763873 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.763837 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pg8c\" (UniqueName: \"kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c\") pod \"637a541c-b238-403e-a12f-a76cd79404cc\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " Jul 30 04:24:11.764052 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.763981 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume\") pod \"637a541c-b238-403e-a12f-a76cd79404cc\" (UID: \"637a541c-b238-403e-a12f-a76cd79404cc\") " Jul 30 04:24:11.764310 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.764288 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "637a541c-b238-403e-a12f-a76cd79404cc" (UID: "637a541c-b238-403e-a12f-a76cd79404cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:24:11.766111 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.766088 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c" (OuterVolumeSpecName: "kube-api-access-5pg8c") pod "637a541c-b238-403e-a12f-a76cd79404cc" (UID: "637a541c-b238-403e-a12f-a76cd79404cc"). InnerVolumeSpecName "kube-api-access-5pg8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:24:11.865703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.865622 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637a541c-b238-403e-a12f-a76cd79404cc-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:24:11.865703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:11.865648 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg8c\" (UniqueName: \"kubernetes.io/projected/637a541c-b238-403e-a12f-a76cd79404cc-kube-api-access-5pg8c\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:24:12.538582 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.538550 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" event={"ID":"637a541c-b238-403e-a12f-a76cd79404cc","Type":"ContainerDied","Data":"ac8cc558b357409d9c8de5025c90dcdbb47b3f01fb9c3908b614579f72881890"} Jul 30 04:24:12.538995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.538604 2568 scope.go:117] "RemoveContainer" containerID="c465685b9a20f7bf36b33e89ed8cebda655f7dccc41f277c2b62b427f7ce125c" Jul 30 04:24:12.538995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.538559 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs" Jul 30 04:24:12.560926 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.560898 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:24:12.575177 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.575148 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" path="/var/lib/kubelet/pods/ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a/volumes" Jul 30 04:24:12.575407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:12.575394 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd1da342-bddf4588d-bqffs"] Jul 30 04:24:14.567408 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:24:14.567368 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637a541c-b238-403e-a12f-a76cd79404cc" path="/var/lib/kubelet/pods/637a541c-b238-403e-a12f-a76cd79404cc/volumes" Jul 30 04:25:02.362617 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:02.362582 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:25:02.363238 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:02.362860 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" containerID="cri-o://acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3" gracePeriod=30 Jul 30 04:25:07.625506 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.625420 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:25:07.723634 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723600 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.723948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723652 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.723948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723682 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.723948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723722 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.723948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723840 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.723948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723930 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.724181 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723965 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.724181 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.723991 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.724711 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.724504 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:25:07.724711 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.724578 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:25:07.724711 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.724680 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:25:07.724986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.724886 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:25:07.724986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.724925 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:25:07.725150 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725027 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpwks\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks\") pod \"72c25e8c-a39e-44b2-9969-353109c2f9d1\" (UID: \"72c25e8c-a39e-44b2-9969-353109c2f9d1\") " Jul 30 04:25:07.725702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725678 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.725837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725705 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.725837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725722 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.725837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725737 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.725837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.725755 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c25e8c-a39e-44b2-9969-353109c2f9d1-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.727800 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.727759 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:25:07.733505 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.733445 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:25:07.735155 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.735129 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks" (OuterVolumeSpecName: "kube-api-access-hpwks") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "kube-api-access-hpwks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:25:07.735226 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.735159 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72c25e8c-a39e-44b2-9969-353109c2f9d1" (UID: "72c25e8c-a39e-44b2-9969-353109c2f9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:25:07.826295 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.826257 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.826295 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.826288 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpwks\" (UniqueName: \"kubernetes.io/projected/72c25e8c-a39e-44b2-9969-353109c2f9d1-kube-api-access-hpwks\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.826295 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.826299 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.826636 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.826308 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c25e8c-a39e-44b2-9969-353109c2f9d1-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:25:07.828928 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.828895 2568 generic.go:356] "Generic (PLEG): container finished" podID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerID="acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3" exitCode=0 Jul 30 04:25:07.829065 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.828976 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" Jul 30 04:25:07.829065 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.828987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerDied","Data":"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3"} Jul 30 04:25:07.829138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.829071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq" event={"ID":"72c25e8c-a39e-44b2-9969-353109c2f9d1","Type":"ContainerDied","Data":"2f076054e2617af8630420ea409371dc8fde579d71a1b9883b55cea3d7adc080"} Jul 30 04:25:07.829138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.829096 2568 scope.go:117] "RemoveContainer" containerID="acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3" Jul 30 04:25:07.839003 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.838985 2568 scope.go:117] "RemoveContainer" containerID="de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e" Jul 30 04:25:07.848238 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.848214 2568 scope.go:117] "RemoveContainer" containerID="acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3" Jul 30 04:25:07.848565 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:25:07.848544 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3\": container with ID starting with acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3 not found: ID does not exist" containerID="acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3" Jul 30 04:25:07.848619 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.848577 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3"} err="failed to get container status \"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3\": rpc error: code = NotFound desc = could not find container \"acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3\": container with ID starting with acc1df1f1e1c656c33064a9d186c2e2419c02d465280e5ada8fbdfb2f3e9c5e3 not found: ID does not exist" Jul 30 04:25:07.848619 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.848597 2568 scope.go:117] "RemoveContainer" containerID="de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e" Jul 30 04:25:07.848920 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:25:07.848899 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e\": container with ID starting with de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e not found: ID does not exist" containerID="de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e" Jul 30 04:25:07.848993 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.848927 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e"} err="failed to get container status \"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e\": rpc error: code = NotFound desc = could not find container \"de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e\": container with ID starting with de6068487f62e36c8c7b62977fbf7536c26332706f0680a2152958f5f86d6f0e not found: ID does not exist" Jul 30 04:25:07.853989 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.853954 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:25:07.859567 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:07.859540 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bd1da342-openshift-default-685766cb8-m5vfq"] Jul 30 04:25:08.567978 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:25:08.567943 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" path="/var/lib/kubelet/pods/72c25e8c-a39e-44b2-9969-353109c2f9d1/volumes" Jul 30 04:26:17.620088 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620047 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620487 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" containerName="revoke-keys" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620502 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" containerName="revoke-keys" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620515 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620521 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620533 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637a541c-b238-403e-a12f-a76cd79404cc" containerName="payload-pre-processing" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620539 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="637a541c-b238-403e-a12f-a76cd79404cc" containerName="payload-pre-processing" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620547 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620552 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620639 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="637a541c-b238-403e-a12f-a76cd79404cc" containerName="payload-pre-processing" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620653 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620667 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c25e8c-a39e-44b2-9969-353109c2f9d1" containerName="istio-proxy" Jul 30 04:26:17.620716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.620678 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca42643a-e5c6-4fcd-956e-9ec6b8e93f0a" containerName="revoke-keys" Jul 30 04:26:17.624338 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.624317 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.626667 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.626646 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9acc4f-openshift-default-dockercfg-6b2kn\"" Jul 30 04:26:17.634824 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.634801 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:26:17.716546 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716504 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmwph\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.716752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.717056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.717056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.716950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817799 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.817995 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.817956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmwph\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818502 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818565 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818630 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818609 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.818921 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.818897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.820448 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.820417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.820589 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.820452 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.825086 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.825063 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.825216 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.825199 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmwph\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph\") pod \"e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:17.939340 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:17.939301 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:18.077063 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.077038 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:26:18.078662 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:26:18.078633 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf902a988_0d0a_4991_a860_e78243b19221.slice/crio-9f4578dbb7be816cc73c5ad77940c077ad49a028082a5523c8bedb54dc371e17 WatchSource:0}: Error finding container 9f4578dbb7be816cc73c5ad77940c077ad49a028082a5523c8bedb54dc371e17: Status 404 returned error can't find the container with id 9f4578dbb7be816cc73c5ad77940c077ad49a028082a5523c8bedb54dc371e17 Jul 30 04:26:18.080860 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.080832 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:18.080940 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.080894 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:18.080940 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.080921 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:18.193298 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.193210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerStarted","Data":"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2"} Jul 30 04:26:18.193298 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.193268 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerStarted","Data":"9f4578dbb7be816cc73c5ad77940c077ad49a028082a5523c8bedb54dc371e17"} Jul 30 04:26:18.216511 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.216435 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podStartSLOduration=1.216412495 podStartE2EDuration="1.216412495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:26:18.212248209 +0000 UTC m=+2458.257747402" watchObservedRunningTime="2026-07-30 04:26:18.216412495 +0000 UTC m=+2458.261911692" Jul 30 04:26:18.939898 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:18.939853 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:19.945109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:19.945078 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:20.202496 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:20.202398 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:20.203562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:20.203540 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:20.768827 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:20.768795 2568 scope.go:117] "RemoveContainer" containerID="65ef3766a08eb57a663fdfbf9200411aedc74087cd25bc6e8feaab89886d527b" Jul 30 04:26:21.844514 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.844475 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:26:21.848125 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.848099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:21.850187 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.850166 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9acc4f\"" Jul 30 04:26:21.850306 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.850284 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9acc4f-dockercfg-pgt6q\"" Jul 30 04:26:21.858708 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.858680 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:26:21.871652 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.871620 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:26:21.874311 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.874292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:21.882645 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.882623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:26:21.957334 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.957297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:21.957522 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.957367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxknh\" (UniqueName: \"kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:21.957522 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.957416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prtpz\" (UniqueName: \"kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:21.957617 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:21.957540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.058562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.058510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:22.058780 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.058606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxknh\" (UniqueName: \"kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.058780 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.058662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prtpz\" (UniqueName: \"kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:22.058780 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.058770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.059478 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.059433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:22.059632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.059611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.066706 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.066677 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxknh\" (UniqueName: \"kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh\") pod \"payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.066845 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.066780 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prtpz\" (UniqueName: \"kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz\") pod \"payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:22.159801 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.159708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:22.186870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.186841 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:22.312168 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.312028 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:26:22.316531 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:26:22.316499 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97b64c1_86b0_4e94_8ed4_cba5fd6ac97b.slice/crio-47c7b3aa6dcf45edd6c17f8c8718654ff4213c2fc1f5e12d450df7c384da11f8 WatchSource:0}: Error finding container 47c7b3aa6dcf45edd6c17f8c8718654ff4213c2fc1f5e12d450df7c384da11f8: Status 404 returned error can't find the container with id 47c7b3aa6dcf45edd6c17f8c8718654ff4213c2fc1f5e12d450df7c384da11f8 Jul 30 04:26:22.347626 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:22.347599 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:26:22.350624 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:26:22.350595 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b217fdb_36d3_4f32_83e7_a1f8f3c1abe2.slice/crio-21f6aed05fc9a20e046b99e363fa9931088d4c3b708afc9c8b3f7396dc172d0e WatchSource:0}: Error finding container 21f6aed05fc9a20e046b99e363fa9931088d4c3b708afc9c8b3f7396dc172d0e: Status 404 returned error can't find the container with id 21f6aed05fc9a20e046b99e363fa9931088d4c3b708afc9c8b3f7396dc172d0e Jul 30 04:26:23.220033 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.219997 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" event={"ID":"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2","Type":"ContainerStarted","Data":"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe"} Jul 30 04:26:23.220667 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.220100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:23.220667 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.220126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" event={"ID":"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2","Type":"ContainerStarted","Data":"21f6aed05fc9a20e046b99e363fa9931088d4c3b708afc9c8b3f7396dc172d0e"} Jul 30 04:26:23.221741 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.221714 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" event={"ID":"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b","Type":"ContainerStarted","Data":"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f"} Jul 30 04:26:23.221853 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.221756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" event={"ID":"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b","Type":"ContainerStarted","Data":"47c7b3aa6dcf45edd6c17f8c8718654ff4213c2fc1f5e12d450df7c384da11f8"} Jul 30 04:26:23.221853 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.221789 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:23.235155 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.235105 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" podStartSLOduration=2.235086426 podStartE2EDuration="2.235086426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:26:23.233715045 +0000 UTC m=+2463.279214237" watchObservedRunningTime="2026-07-30 04:26:23.235086426 +0000 UTC m=+2463.280585624" Jul 30 04:26:23.246804 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:23.246748 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" podStartSLOduration=2.246732307 podStartE2EDuration="2.246732307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:26:23.24665678 +0000 UTC m=+2463.292155977" watchObservedRunningTime="2026-07-30 04:26:23.246732307 +0000 UTC m=+2463.292231484" Jul 30 04:26:34.230233 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.230194 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:26:34.230702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.230275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:26:34.294345 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.294267 2568 generic.go:356] "Generic (PLEG): container finished" podID="f902a988-0d0a-4991-a860-e78243b19221" containerID="da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2" exitCode=0 Jul 30 04:26:34.294521 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.294350 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerDied","Data":"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2"} Jul 30 04:26:34.294787 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.294773 2568 scope.go:117] "RemoveContainer" containerID="da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2" Jul 30 04:26:34.295815 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.295787 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:34.295907 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.295868 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:34.295992 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:34.295917 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:35.203331 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:35.203290 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:35.301553 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:35.301511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerStarted","Data":"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8"} Jul 30 04:26:35.940109 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:35.940071 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:35.941328 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:35.941294 2568 patch_prober.go:28] interesting pod/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 30 04:26:35.941460 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:35.941364 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 30 04:26:36.940233 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:36.940200 2568 patch_prober.go:28] interesting pod/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 30 04:26:36.940672 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:36.940254 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 30 04:26:37.940481 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:37.940435 2568 patch_prober.go:28] interesting pod/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 30 04:26:37.940889 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:37.940526 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 30 04:26:38.939737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:38.939703 2568 patch_prober.go:28] interesting pod/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 30 04:26:38.939923 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:38.939758 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 30 04:26:39.944323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:39.944285 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:39.944781 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:39.944613 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:39.945330 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:39.945312 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:26:45.816169 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.816135 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:26:45.820784 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.820751 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.823664 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.823641 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c24378-openshift-default-dockercfg-frbgr\"" Jul 30 04:26:45.833326 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.833299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:26:45.910816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.910771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911126 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911073 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911126 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911330 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911330 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911330 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911268 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6lv\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911330 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911595 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911354 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:45.911595 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:45.911386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012152 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012152 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6lv\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012839 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012697 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012877 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.012919 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.012867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.013058 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.013043 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.014949 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.014924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.015113 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.015097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.020184 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.020158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.020447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.020421 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6lv\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv\") pod \"e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.134055 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.133972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:46.278097 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.278072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:26:46.280058 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:26:46.280032 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766d103b_b2bf_444f_a1c9_0da1629a9f97.slice/crio-66e84a63b2d779c365dbcd466cfe0f2810695e1c0f4113cfdf9f616f1875515e WatchSource:0}: Error finding container 66e84a63b2d779c365dbcd466cfe0f2810695e1c0f4113cfdf9f616f1875515e: Status 404 returned error can't find the container with id 66e84a63b2d779c365dbcd466cfe0f2810695e1c0f4113cfdf9f616f1875515e Jul 30 04:26:46.282374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.282335 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:46.282455 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.282433 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:46.282537 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.282509 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:26:46.371948 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.371912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerStarted","Data":"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3"} Jul 30 04:26:46.372136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.371976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerStarted","Data":"66e84a63b2d779c365dbcd466cfe0f2810695e1c0f4113cfdf9f616f1875515e"} Jul 30 04:26:46.392850 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:46.392735 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podStartSLOduration=1.392714453 podStartE2EDuration="1.392714453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:26:46.391614391 +0000 UTC m=+2486.437113621" watchObservedRunningTime="2026-07-30 04:26:46.392714453 +0000 UTC m=+2486.438213649" Jul 30 04:26:47.134510 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:47.134455 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:47.139348 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:47.139323 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:47.377419 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:47.377390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:26:47.378529 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:26:47.378505 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:27:03.274999 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.274438 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:48818->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 30 04:27:03.274999 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.274568 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:48818->10.134.0.56:15021: read: connection reset by peer" Jul 30 04:27:03.469839 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.469736 2568 generic.go:356] "Generic (PLEG): container finished" podID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerID="d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3" exitCode=0 Jul 30 04:27:03.469839 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.469809 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerDied","Data":"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3"} Jul 30 04:27:03.470318 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.470303 2568 scope.go:117] "RemoveContainer" containerID="d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3" Jul 30 04:27:03.471434 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.471405 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:27:03.471562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.471492 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:27:03.471562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:03.471531 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:27:04.478032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:04.477986 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerStarted","Data":"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2"} Jul 30 04:27:05.134599 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:05.134552 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:27:05.134901 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:05.134793 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 30 04:27:05.134901 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:05.134846 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 30 04:27:06.134540 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:06.134506 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 30 04:27:06.134935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:06.134574 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 30 04:27:07.134303 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:07.134267 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 30 04:27:07.134514 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:07.134337 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 30 04:27:08.134669 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:08.134625 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 30 04:27:08.135075 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:08.134698 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 30 04:27:09.138835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:09.138802 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:27:09.139276 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:09.139144 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:27:09.139895 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:27:09.139874 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:28:28.838860 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.838821 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp"] Jul 30 04:28:28.841512 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.841491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:28.843587 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.843567 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:28:28.843689 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.843667 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:28:28.849087 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.849056 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp"] Jul 30 04:28:28.862256 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.862226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp\" (UID: \"4b6a43ab-c519-4753-98c6-277e2a2ccfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:28.963842 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.963798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp\" (UID: \"4b6a43ab-c519-4753-98c6-277e2a2ccfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:28.964416 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:28.964397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp\" (UID: \"4b6a43ab-c519-4753-98c6-277e2a2ccfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:29.153842 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:29.153747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:29.300544 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:29.300512 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp"] Jul 30 04:28:29.303192 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:28:29.303160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6a43ab_c519_4753_98c6_277e2a2ccfb3.slice/crio-d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072 WatchSource:0}: Error finding container d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072: Status 404 returned error can't find the container with id d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072 Jul 30 04:28:29.918534 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:29.918497 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" event={"ID":"4b6a43ab-c519-4753-98c6-277e2a2ccfb3","Type":"ContainerStarted","Data":"e3e579a1896c4cc58d32b5083ee189d86655ee28cda2fc3775d39deea2c46567"} Jul 30 04:28:29.919017 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:29.918567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" event={"ID":"4b6a43ab-c519-4753-98c6-277e2a2ccfb3","Type":"ContainerStarted","Data":"d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072"} Jul 30 04:28:29.933364 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:29.933277 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" podStartSLOduration=1.9332595160000001 podStartE2EDuration="1.933259516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:28:29.931775717 +0000 UTC m=+2589.977274913" watchObservedRunningTime="2026-07-30 04:28:29.933259516 +0000 UTC m=+2589.978758711" Jul 30 04:28:30.926051 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:30.926004 2568 generic.go:356] "Generic (PLEG): container finished" podID="4b6a43ab-c519-4753-98c6-277e2a2ccfb3" containerID="e3e579a1896c4cc58d32b5083ee189d86655ee28cda2fc3775d39deea2c46567" exitCode=0 Jul 30 04:28:30.926441 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:30.926086 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" event={"ID":"4b6a43ab-c519-4753-98c6-277e2a2ccfb3","Type":"ContainerDied","Data":"e3e579a1896c4cc58d32b5083ee189d86655ee28cda2fc3775d39deea2c46567"} Jul 30 04:28:32.075522 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.075450 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:32.097276 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.097249 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca\") pod \"4b6a43ab-c519-4753-98c6-277e2a2ccfb3\" (UID: \"4b6a43ab-c519-4753-98c6-277e2a2ccfb3\") " Jul 30 04:28:32.097626 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.097601 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b6a43ab-c519-4753-98c6-277e2a2ccfb3" (UID: "4b6a43ab-c519-4753-98c6-277e2a2ccfb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:28:32.198728 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.198645 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a43ab-c519-4753-98c6-277e2a2ccfb3-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:32.937684 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.937658 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" Jul 30 04:28:32.937859 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.937657 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp" event={"ID":"4b6a43ab-c519-4753-98c6-277e2a2ccfb3","Type":"ContainerDied","Data":"d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072"} Jul 30 04:28:32.937859 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:32.937782 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f5d3b7b5ba5f6cf50234f910171e99847a1a8f93a9b11574fbe5f96987b072" Jul 30 04:28:34.919716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:34.919679 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:28:34.920205 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:34.920140 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" podUID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" containerName="payload-processing" containerID="cri-o://a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe" gracePeriod=30 Jul 30 04:28:34.927345 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:34.927300 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:28:34.927684 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:34.927607 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" podUID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" containerName="payload-pre-processing" containerID="cri-o://eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f" gracePeriod=30 Jul 30 04:28:35.195218 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.195183 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp"] Jul 30 04:28:35.199382 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.199355 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9acc4f-vwtgp"] Jul 30 04:28:35.209209 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.209184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:28:35.235012 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.234987 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume\") pod \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " Jul 30 04:28:35.235149 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.235078 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prtpz\" (UniqueName: \"kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz\") pod \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\" (UID: \"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2\") " Jul 30 04:28:35.235416 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.235396 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" (UID: "6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:28:35.237343 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.237315 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz" (OuterVolumeSpecName: "kube-api-access-prtpz") pod "6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" (UID: "6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2"). InnerVolumeSpecName "kube-api-access-prtpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:28:35.239832 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.239814 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:28:35.336402 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.336363 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxknh\" (UniqueName: \"kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh\") pod \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " Jul 30 04:28:35.336674 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.336538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume\") pod \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\" (UID: \"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b\") " Jul 30 04:28:35.336923 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.336900 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" (UID: "f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:28:35.337009 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.336972 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prtpz\" (UniqueName: \"kubernetes.io/projected/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-kube-api-access-prtpz\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:35.337009 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.336994 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:35.338696 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.338671 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh" (OuterVolumeSpecName: "kube-api-access-nxknh") pod "f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" (UID: "f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b"). InnerVolumeSpecName "kube-api-access-nxknh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:28:35.438444 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.438352 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:35.438444 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.438383 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxknh\" (UniqueName: \"kubernetes.io/projected/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b-kube-api-access-nxknh\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:35.957142 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.957106 2568 generic.go:356] "Generic (PLEG): container finished" podID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" containerID="a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe" exitCode=0 Jul 30 04:28:35.957632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.957149 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" Jul 30 04:28:35.957632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.957185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" event={"ID":"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2","Type":"ContainerDied","Data":"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe"} Jul 30 04:28:35.957632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.957242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw" event={"ID":"6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2","Type":"ContainerDied","Data":"21f6aed05fc9a20e046b99e363fa9931088d4c3b708afc9c8b3f7396dc172d0e"} Jul 30 04:28:35.957632 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.957264 2568 scope.go:117] "RemoveContainer" containerID="a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe" Jul 30 04:28:35.958717 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.958696 2568 generic.go:356] "Generic (PLEG): container finished" podID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" containerID="eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f" exitCode=0 Jul 30 04:28:35.958795 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.958748 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" Jul 30 04:28:35.958795 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.958770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" event={"ID":"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b","Type":"ContainerDied","Data":"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f"} Jul 30 04:28:35.958884 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.958815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc" event={"ID":"f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b","Type":"ContainerDied","Data":"47c7b3aa6dcf45edd6c17f8c8718654ff4213c2fc1f5e12d450df7c384da11f8"} Jul 30 04:28:35.967316 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.967291 2568 scope.go:117] "RemoveContainer" containerID="a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe" Jul 30 04:28:35.967639 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:28:35.967619 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe\": container with ID starting with a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe not found: ID does not exist" containerID="a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe" Jul 30 04:28:35.967723 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.967648 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe"} err="failed to get container status \"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe\": rpc error: code = NotFound desc = could not find container \"a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe\": container with ID starting with a5493753abbc56fa70065e6addbf7d15d97707f8222cddb4e6beccfe06106ffe not found: ID does not exist" Jul 30 04:28:35.967723 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.967666 2568 scope.go:117] "RemoveContainer" containerID="eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f" Jul 30 04:28:35.976875 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.976858 2568 scope.go:117] "RemoveContainer" containerID="eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f" Jul 30 04:28:35.977167 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:28:35.977147 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f\": container with ID starting with eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f not found: ID does not exist" containerID="eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f" Jul 30 04:28:35.977214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.977175 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f"} err="failed to get container status \"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f\": rpc error: code = NotFound desc = could not find container \"eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f\": container with ID starting with eb620b77aa7f61642a024125ae3308a3ad6807cef73cac22d447ce37ea9af56f not found: ID does not exist" Jul 30 04:28:35.995392 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:35.995356 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:28:36.014325 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.014292 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9acc4f-c86d78564-6vzkc"] Jul 30 04:28:36.046574 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.046524 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:28:36.048868 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.048843 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9acc4f-75998c5d4b-b7blw"] Jul 30 04:28:36.567835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.567800 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6a43ab-c519-4753-98c6-277e2a2ccfb3" path="/var/lib/kubelet/pods/4b6a43ab-c519-4753-98c6-277e2a2ccfb3/volumes" Jul 30 04:28:36.568222 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.568209 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" path="/var/lib/kubelet/pods/6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2/volumes" Jul 30 04:28:36.568614 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:36.568600 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" path="/var/lib/kubelet/pods/f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b/volumes" Jul 30 04:28:46.677425 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:46.677384 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:28:46.677928 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:46.677648 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" containerID="cri-o://fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8" gracePeriod=30 Jul 30 04:28:47.055612 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.055527 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp"] Jul 30 04:28:47.055975 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.055960 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" containerName="payload-pre-processing" Jul 30 04:28:47.056024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.055979 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" containerName="payload-pre-processing" Jul 30 04:28:47.056024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.055992 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6a43ab-c519-4753-98c6-277e2a2ccfb3" containerName="revoke-keys" Jul 30 04:28:47.056024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.055998 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6a43ab-c519-4753-98c6-277e2a2ccfb3" containerName="revoke-keys" Jul 30 04:28:47.056024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.056014 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" containerName="payload-processing" Jul 30 04:28:47.056024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.056019 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" containerName="payload-processing" Jul 30 04:28:47.056207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.056082 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97b64c1-86b0-4e94-8ed4-cba5fd6ac97b" containerName="payload-pre-processing" Jul 30 04:28:47.056207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.056092 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b217fdb-36d3-4f32-83e7-a1f8f3c1abe2" containerName="payload-processing" Jul 30 04:28:47.056207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.056101 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6a43ab-c519-4753-98c6-277e2a2ccfb3" containerName="revoke-keys" Jul 30 04:28:47.059613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.059591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:47.061928 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.061907 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:28:47.062052 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.061907 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:28:47.066200 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.066155 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp"] Jul 30 04:28:47.157386 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.157343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24378-55ztp\" (UID: \"fd5cb90e-9d2f-4787-98c6-b65881c108e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:47.258573 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.258538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24378-55ztp\" (UID: \"fd5cb90e-9d2f-4787-98c6-b65881c108e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:47.259256 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.259227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24378-55ztp\" (UID: \"fd5cb90e-9d2f-4787-98c6-b65881c108e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:47.372913 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.372819 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:47.517565 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:47.517530 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp"] Jul 30 04:28:47.519310 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:28:47.519270 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5cb90e_9d2f_4787_98c6_b65881c108e9.slice/crio-9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174 WatchSource:0}: Error finding container 9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174: Status 404 returned error can't find the container with id 9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174 Jul 30 04:28:48.024323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:48.024225 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" event={"ID":"fd5cb90e-9d2f-4787-98c6-b65881c108e9","Type":"ContainerStarted","Data":"1c2c8fcd29a5a97f0d2e1a91b3dc28422f9845a593491d3f92a404caa6709488"} Jul 30 04:28:48.024323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:48.024292 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" event={"ID":"fd5cb90e-9d2f-4787-98c6-b65881c108e9","Type":"ContainerStarted","Data":"9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174"} Jul 30 04:28:48.039147 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:48.039097 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" podStartSLOduration=1.03908167 podStartE2EDuration="1.03908167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:28:48.036727172 +0000 UTC m=+2608.082226382" watchObservedRunningTime="2026-07-30 04:28:48.03908167 +0000 UTC m=+2608.084580919" Jul 30 04:28:49.030859 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:49.030826 2568 generic.go:356] "Generic (PLEG): container finished" podID="fd5cb90e-9d2f-4787-98c6-b65881c108e9" containerID="1c2c8fcd29a5a97f0d2e1a91b3dc28422f9845a593491d3f92a404caa6709488" exitCode=0 Jul 30 04:28:49.031262 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:49.030913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" event={"ID":"fd5cb90e-9d2f-4787-98c6-b65881c108e9","Type":"ContainerDied","Data":"1c2c8fcd29a5a97f0d2e1a91b3dc28422f9845a593491d3f92a404caa6709488"} Jul 30 04:28:50.173389 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:50.173365 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:50.291066 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:50.290969 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca\") pod \"fd5cb90e-9d2f-4787-98c6-b65881c108e9\" (UID: \"fd5cb90e-9d2f-4787-98c6-b65881c108e9\") " Jul 30 04:28:50.291365 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:50.291338 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd5cb90e-9d2f-4787-98c6-b65881c108e9" (UID: "fd5cb90e-9d2f-4787-98c6-b65881c108e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:28:50.392044 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:50.392007 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5cb90e-9d2f-4787-98c6-b65881c108e9-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:51.043114 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:51.043086 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" Jul 30 04:28:51.043294 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:51.043130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp" event={"ID":"fd5cb90e-9d2f-4787-98c6-b65881c108e9","Type":"ContainerDied","Data":"9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174"} Jul 30 04:28:51.043294 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:51.043176 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f999f6962ca512143994b5d8257aa73bd275d60dfe1605230d16967e7a67174" Jul 30 04:28:51.925626 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:51.925598 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:28:52.008874 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.008765 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.008874 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.008855 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.008901 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.008919 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.008960 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009001 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009053 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009080 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmwph\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009121 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009107 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy\") pod \"f902a988-0d0a-4991-a860-e78243b19221\" (UID: \"f902a988-0d0a-4991-a860-e78243b19221\") " Jul 30 04:28:52.009494 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009214 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:28:52.009494 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009230 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:28:52.009494 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009301 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:28:52.009494 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009325 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:28:52.009946 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009579 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.009946 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009595 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f902a988-0d0a-4991-a860-e78243b19221-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.009946 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009603 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.009946 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.009612 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.010165 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.010132 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data" (OuterVolumeSpecName: "istio-data") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:28:52.011344 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.011324 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:28:52.011817 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.011794 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:28:52.011922 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.011809 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token" (OuterVolumeSpecName: "istio-token") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:28:52.012122 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.012100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph" (OuterVolumeSpecName: "kube-api-access-vmwph") pod "f902a988-0d0a-4991-a860-e78243b19221" (UID: "f902a988-0d0a-4991-a860-e78243b19221"). InnerVolumeSpecName "kube-api-access-vmwph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:28:52.050128 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.050091 2568 generic.go:356] "Generic (PLEG): container finished" podID="f902a988-0d0a-4991-a860-e78243b19221" containerID="fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8" exitCode=0 Jul 30 04:28:52.050273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.050135 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" Jul 30 04:28:52.050273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.050152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerDied","Data":"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8"} Jul 30 04:28:52.050273 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.050231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b" event={"ID":"f902a988-0d0a-4991-a860-e78243b19221","Type":"ContainerDied","Data":"9f4578dbb7be816cc73c5ad77940c077ad49a028082a5523c8bedb54dc371e17"} Jul 30 04:28:52.050410 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.050255 2568 scope.go:117] "RemoveContainer" containerID="fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8" Jul 30 04:28:52.060709 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.060690 2568 scope.go:117] "RemoveContainer" containerID="da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2" Jul 30 04:28:52.068849 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.068826 2568 scope.go:117] "RemoveContainer" containerID="fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8" Jul 30 04:28:52.069442 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:28:52.069217 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8\": container with ID starting with fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8 not found: ID does not exist" containerID="fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8" Jul 30 04:28:52.069442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.069257 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8"} err="failed to get container status \"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8\": rpc error: code = NotFound desc = could not find container \"fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8\": container with ID starting with fbc0a6bc022173fc84a2e50f193e048aef17d17f8f281d8676039d9f078b3db8 not found: ID does not exist" Jul 30 04:28:52.069442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.069282 2568 scope.go:117] "RemoveContainer" containerID="da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2" Jul 30 04:28:52.069797 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:28:52.069557 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2\": container with ID starting with da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2 not found: ID does not exist" containerID="da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2" Jul 30 04:28:52.069797 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.069592 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2"} err="failed to get container status \"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2\": rpc error: code = NotFound desc = could not find container \"da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2\": container with ID starting with da693138a656a80a605d1462d9c3151e5151a0737f46c8f27b8d2972fb7751a2 not found: ID does not exist" Jul 30 04:28:52.071835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.071815 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:28:52.075030 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.075008 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9acc4f-openshift-default-57854b94fd-pd24b"] Jul 30 04:28:52.110185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.110151 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.110185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.110177 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.110185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.110188 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmwph\" (UniqueName: \"kubernetes.io/projected/f902a988-0d0a-4991-a860-e78243b19221-kube-api-access-vmwph\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.110390 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.110197 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f902a988-0d0a-4991-a860-e78243b19221-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.110390 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.110206 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f902a988-0d0a-4991-a860-e78243b19221-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:28:52.570904 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:52.570867 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f902a988-0d0a-4991-a860-e78243b19221" path="/var/lib/kubelet/pods/f902a988-0d0a-4991-a860-e78243b19221/volumes" Jul 30 04:28:57.676384 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:57.676346 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp"] Jul 30 04:28:57.678836 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:57.678806 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24378-55ztp"] Jul 30 04:28:58.568847 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:28:58.568811 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5cb90e-9d2f-4787-98c6-b65881c108e9" path="/var/lib/kubelet/pods/fd5cb90e-9d2f-4787-98c6-b65881c108e9/volumes" Jul 30 04:29:04.207532 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:04.207494 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:29:04.208053 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:04.207763 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" containerID="cri-o://3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2" gracePeriod=30 Jul 30 04:29:05.082137 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082103 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:29:05.082574 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082559 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:05.082574 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082574 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:05.082702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082584 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5cb90e-9d2f-4787-98c6-b65881c108e9" containerName="revoke-keys" Jul 30 04:29:05.082702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082590 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5cb90e-9d2f-4787-98c6-b65881c108e9" containerName="revoke-keys" Jul 30 04:29:05.082702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082673 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:05.082702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.082690 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5cb90e-9d2f-4787-98c6-b65881c108e9" containerName="revoke-keys" Jul 30 04:29:05.085092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.085074 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.087506 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.087481 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eb6c99-openshift-default-dockercfg-nh6tw\"" Jul 30 04:29:05.098493 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.098446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:29:05.144706 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.144706 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144710 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.144961 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsppc\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.144961 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144813 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.144961 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.144961 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.145129 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.144980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.145129 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.145008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.145129 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.145031 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.245800 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245766 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsppc\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.245986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246052 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246315 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246283 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.246716 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.246647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.247103 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.247081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.248549 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.248531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.248834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.248815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.254564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.254535 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.255049 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.255027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsppc\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc\") pod \"e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.395837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.395725 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:05.541860 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.541829 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:29:05.545197 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:29:05.545172 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf529d16e_0992_47e7_9944_c207fb4b20cc.slice/crio-c219e8e020a681a04b23723d3756b808f83632181823aba52be89a5387205b64 WatchSource:0}: Error finding container c219e8e020a681a04b23723d3756b808f83632181823aba52be89a5387205b64: Status 404 returned error can't find the container with id c219e8e020a681a04b23723d3756b808f83632181823aba52be89a5387205b64 Jul 30 04:29:05.547843 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.547800 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:05.547952 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.547881 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:05.547952 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:05.547923 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:06.127638 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:06.127593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerStarted","Data":"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44"} Jul 30 04:29:06.127828 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:06.127661 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerStarted","Data":"c219e8e020a681a04b23723d3756b808f83632181823aba52be89a5387205b64"} Jul 30 04:29:06.148942 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:06.148884 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podStartSLOduration=1.148866397 podStartE2EDuration="1.148866397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:29:06.146035942 +0000 UTC m=+2626.191535159" watchObservedRunningTime="2026-07-30 04:29:06.148866397 +0000 UTC m=+2626.194365591" Jul 30 04:29:06.396789 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:06.396693 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:06.401619 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:06.401592 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:07.132566 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:07.132538 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:07.133521 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:07.133504 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:09.141540 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.141502 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 04:29:09.141967 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.141578 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 04:29:09.462216 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.462191 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:29:09.480997 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.480968 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.481207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481007 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.481207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481034 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.481207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481081 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.481207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481122 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.481207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481169 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481298 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481358 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.481436 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482158 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data" (OuterVolumeSpecName: "istio-data") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482227 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r6lv\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv\") pod \"766d103b-b2bf-444f-a1c9-0da1629a9f97\" (UID: \"766d103b-b2bf-444f-a1c9-0da1629a9f97\") " Jul 30 04:29:09.482518 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482267 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:29:09.482927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482603 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.482927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482625 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.482927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482639 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.482927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482650 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.482927 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.482695 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:29:09.484920 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.484763 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:29:09.485354 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.485328 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:29:09.485880 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.485855 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv" (OuterVolumeSpecName: "kube-api-access-5r6lv") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "kube-api-access-5r6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:29:09.485880 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.485869 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token" (OuterVolumeSpecName: "istio-token") pod "766d103b-b2bf-444f-a1c9-0da1629a9f97" (UID: "766d103b-b2bf-444f-a1c9-0da1629a9f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:29:09.583542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.583509 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.583542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.583537 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6lv\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-kube-api-access-5r6lv\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.583542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.583546 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.583790 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.583555 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d103b-b2bf-444f-a1c9-0da1629a9f97-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.583790 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.583565 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d103b-b2bf-444f-a1c9-0da1629a9f97-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:29:09.853020 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.852930 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:29:09.853383 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853369 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:09.853436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853386 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:09.853436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853402 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:09.853436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853408 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:09.853577 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853495 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:09.853577 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853504 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f902a988-0d0a-4991-a860-e78243b19221" containerName="istio-proxy" Jul 30 04:29:09.853577 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.853510 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:09.856198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.856181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.858532 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.858507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eb6c99\"" Jul 30 04:29:09.858731 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.858710 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eb6c99-dockercfg-b9mps\"" Jul 30 04:29:09.864573 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.864547 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:29:09.886226 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.886192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.886427 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.886303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6l5\" (UniqueName: \"kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.987022 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.986980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.987217 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.987069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6l5\" (UniqueName: \"kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.987722 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.987702 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:09.994915 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:09.994887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6l5\" (UniqueName: \"kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5\") pod \"payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:10.153374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.153272 2568 generic.go:356] "Generic (PLEG): container finished" podID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerID="3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2" exitCode=0 Jul 30 04:29:10.153374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.153320 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" Jul 30 04:29:10.153374 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.153361 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerDied","Data":"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2"} Jul 30 04:29:10.153945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.153427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv" event={"ID":"766d103b-b2bf-444f-a1c9-0da1629a9f97","Type":"ContainerDied","Data":"66e84a63b2d779c365dbcd466cfe0f2810695e1c0f4113cfdf9f616f1875515e"} Jul 30 04:29:10.153945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.153493 2568 scope.go:117] "RemoveContainer" containerID="3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2" Jul 30 04:29:10.163718 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.163700 2568 scope.go:117] "RemoveContainer" containerID="d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3" Jul 30 04:29:10.168918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.168895 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:10.173333 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.173098 2568 scope.go:117] "RemoveContainer" containerID="3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2" Jul 30 04:29:10.173730 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:29:10.173702 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2\": container with ID starting with 3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2 not found: ID does not exist" containerID="3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2" Jul 30 04:29:10.173825 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.173745 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2"} err="failed to get container status \"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2\": rpc error: code = NotFound desc = could not find container \"3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2\": container with ID starting with 3ed226cced58af10fe933e9adb6df217cc999df5cccb68b6e625e4daf340dde2 not found: ID does not exist" Jul 30 04:29:10.173825 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.173770 2568 scope.go:117] "RemoveContainer" containerID="d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3" Jul 30 04:29:10.174082 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:29:10.174055 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3\": container with ID starting with d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3 not found: ID does not exist" containerID="d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3" Jul 30 04:29:10.174138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.174094 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3"} err="failed to get container status \"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3\": rpc error: code = NotFound desc = could not find container \"d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3\": container with ID starting with d0aa47c06aa219887ee644c528ceae35fa33d73fc1852fe83bad84bb0946f7c3 not found: ID does not exist" Jul 30 04:29:10.175164 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.175145 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:29:10.178637 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.178616 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c24378-openshift-default-7f64556698-2lnjv"] Jul 30 04:29:10.306764 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.306740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:29:10.309586 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:29:10.309558 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9e6c18_e65d_42e0_b566_c409ebe83398.slice/crio-810ddcff58ee210e925cff53009747c10e8eabc90f158274aa0b425b765ef3c2 WatchSource:0}: Error finding container 810ddcff58ee210e925cff53009747c10e8eabc90f158274aa0b425b765ef3c2: Status 404 returned error can't find the container with id 810ddcff58ee210e925cff53009747c10e8eabc90f158274aa0b425b765ef3c2 Jul 30 04:29:10.569421 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:10.569389 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" path="/var/lib/kubelet/pods/766d103b-b2bf-444f-a1c9-0da1629a9f97/volumes" Jul 30 04:29:11.160886 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:11.160847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" event={"ID":"2d9e6c18-e65d-42e0-b566-c409ebe83398","Type":"ContainerStarted","Data":"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1"} Jul 30 04:29:11.161407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:11.160946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" event={"ID":"2d9e6c18-e65d-42e0-b566-c409ebe83398","Type":"ContainerStarted","Data":"810ddcff58ee210e925cff53009747c10e8eabc90f158274aa0b425b765ef3c2"} Jul 30 04:29:11.161407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:11.160977 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:11.176933 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:11.176864 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" podStartSLOduration=2.176847258 podStartE2EDuration="2.176847258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:29:11.174622312 +0000 UTC m=+2631.220121508" watchObservedRunningTime="2026-07-30 04:29:11.176847258 +0000 UTC m=+2631.222346483" Jul 30 04:29:20.216665 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.216633 2568 generic.go:356] "Generic (PLEG): container finished" podID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerID="86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44" exitCode=0 Jul 30 04:29:20.217102 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.216709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerDied","Data":"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44"} Jul 30 04:29:20.217154 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.217135 2568 scope.go:117] "RemoveContainer" containerID="86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44" Jul 30 04:29:20.218251 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.218223 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:20.218321 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.218305 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:20.218370 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:20.218357 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:21.223684 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:21.223647 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerStarted","Data":"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39"} Jul 30 04:29:21.396785 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:21.396749 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:21.397997 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:21.397968 2568 patch_prober.go:28] interesting pod/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 30 04:29:21.398152 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:21.398041 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 30 04:29:22.132845 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:22.132808 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:22.168665 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:22.168632 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:29:22.396917 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:22.396831 2568 patch_prober.go:28] interesting pod/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 30 04:29:22.396917 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:22.396883 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 30 04:29:23.396682 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:23.396642 2568 patch_prober.go:28] interesting pod/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 30 04:29:23.396880 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:23.396699 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 30 04:29:24.397086 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:24.397045 2568 patch_prober.go:28] interesting pod/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 30 04:29:24.397585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:24.397109 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 30 04:29:25.399954 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:25.399921 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:25.401116 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:25.401094 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:29:33.806677 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.806641 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:29:33.807342 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.807317 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:33.807342 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.807342 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="766d103b-b2bf-444f-a1c9-0da1629a9f97" containerName="istio-proxy" Jul 30 04:29:33.809773 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.809753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.812091 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.812067 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-587da0-openshift-default-dockercfg-7g9nx\"" Jul 30 04:29:33.820869 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.820846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:29:33.929290 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929502 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929502 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929502 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929520 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929755 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:33.929755 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:33.929701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bpq\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031319 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bpq\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.031585 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032022 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032022 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031871 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032022 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.031965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032346 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.032137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032346 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.032164 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.032483 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.032431 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.033939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.033916 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.034270 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.034248 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.039073 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.039048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.039244 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.039226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bpq\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq\") pod \"e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.121536 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.121437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:34.267179 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.267147 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:29:34.270497 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:29:34.270454 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23a77da_ec89_4b61_8a0e_4f18ec7e17d0.slice/crio-af07e07761d1b13c0dde9bd6cf8f6d52eed34981199023adc97ca4ba2f63590c WatchSource:0}: Error finding container af07e07761d1b13c0dde9bd6cf8f6d52eed34981199023adc97ca4ba2f63590c: Status 404 returned error can't find the container with id af07e07761d1b13c0dde9bd6cf8f6d52eed34981199023adc97ca4ba2f63590c Jul 30 04:29:34.272828 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.272797 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:34.272936 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.272873 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:34.272936 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.272920 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:34.293161 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:34.293133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerStarted","Data":"af07e07761d1b13c0dde9bd6cf8f6d52eed34981199023adc97ca4ba2f63590c"} Jul 30 04:29:35.300863 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:35.300820 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerStarted","Data":"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae"} Jul 30 04:29:35.324490 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:35.323439 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podStartSLOduration=2.323406049 podStartE2EDuration="2.323406049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:29:35.319321115 +0000 UTC m=+2655.364820310" watchObservedRunningTime="2026-07-30 04:29:35.323406049 +0000 UTC m=+2655.368905245" Jul 30 04:29:36.122284 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:36.122250 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:36.127272 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:36.127245 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:36.310040 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:36.310011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:36.311860 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:36.311841 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:39.088160 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.088118 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:29:39.090945 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.090929 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.093275 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.093252 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-587da0\"" Jul 30 04:29:39.093275 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.093267 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-587da0-dockercfg-qfgq8\"" Jul 30 04:29:39.106795 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.106769 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:29:39.186258 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.186219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.186500 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.186367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.287348 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.287313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.287558 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.287392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.288056 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.288035 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.294983 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.294957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl\") pod \"payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.402928 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.402826 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:39.547531 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:39.547503 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:29:39.549215 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:29:39.549184 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386dfdd3_99d2_4d5a_8d0c_2fb3ff02ee79.slice/crio-7ff4eda8e45d6237f93bfc12ab2a80279c07bf4bc46cebd587f28cab2486ca1e WatchSource:0}: Error finding container 7ff4eda8e45d6237f93bfc12ab2a80279c07bf4bc46cebd587f28cab2486ca1e: Status 404 returned error can't find the container with id 7ff4eda8e45d6237f93bfc12ab2a80279c07bf4bc46cebd587f28cab2486ca1e Jul 30 04:29:40.333058 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:40.333018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" event={"ID":"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79","Type":"ContainerStarted","Data":"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327"} Jul 30 04:29:40.333488 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:40.333124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" event={"ID":"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79","Type":"ContainerStarted","Data":"7ff4eda8e45d6237f93bfc12ab2a80279c07bf4bc46cebd587f28cab2486ca1e"} Jul 30 04:29:40.333488 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:40.333158 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:40.350810 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:40.350755 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" podStartSLOduration=1.350738384 podStartE2EDuration="1.350738384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:29:40.348931586 +0000 UTC m=+2660.394430782" watchObservedRunningTime="2026-07-30 04:29:40.350738384 +0000 UTC m=+2660.396237578" Jul 30 04:29:51.310850 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.310808 2568 patch_prober.go:28] interesting pod/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 30 04:29:51.311338 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.310873 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 30 04:29:51.340343 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.340314 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:29:51.403089 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.403057 2568 generic.go:356] "Generic (PLEG): container finished" podID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerID="5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae" exitCode=0 Jul 30 04:29:51.403260 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.403143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerDied","Data":"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae"} Jul 30 04:29:51.403743 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.403721 2568 scope.go:117] "RemoveContainer" containerID="5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae" Jul 30 04:29:51.404896 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.404865 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:51.404983 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.404956 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:51.405027 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:51.405001 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:29:52.411075 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:52.411037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerStarted","Data":"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe"} Jul 30 04:29:53.122088 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:53.122059 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:53.122447 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:53.122420 2568 patch_prober.go:28] interesting pod/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 30 04:29:53.122583 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:53.122489 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 30 04:29:54.122042 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:54.122003 2568 patch_prober.go:28] interesting pod/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 30 04:29:54.122436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:54.122072 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 30 04:29:55.122739 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:55.122698 2568 patch_prober.go:28] interesting pod/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 30 04:29:55.123136 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:55.122755 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 30 04:29:56.122628 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:56.122584 2568 patch_prober.go:28] interesting pod/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 30 04:29:56.122821 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:56.122657 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 30 04:29:57.127151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:57.127121 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:57.127562 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:57.127497 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:29:57.128248 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:29:57.128217 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:30:20.842430 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:20.842341 2568 scope.go:117] "RemoveContainer" containerID="097e6b88f3ed86c3b0f2b4e833493de7c514b9a0228b26fe885a9839e76d6c59" Jul 30 04:30:20.874652 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:20.874610 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:30:20.874980 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:20.874894 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" podUID="2d9e6c18-e65d-42e0-b566-c409ebe83398" containerName="payload-pre-processing" containerID="cri-o://56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1" gracePeriod=30 Jul 30 04:30:21.127192 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.127167 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:30:21.221507 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.221457 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq6l5\" (UniqueName: \"kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5\") pod \"2d9e6c18-e65d-42e0-b566-c409ebe83398\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " Jul 30 04:30:21.221759 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.221552 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume\") pod \"2d9e6c18-e65d-42e0-b566-c409ebe83398\" (UID: \"2d9e6c18-e65d-42e0-b566-c409ebe83398\") " Jul 30 04:30:21.221942 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.221914 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d9e6c18-e65d-42e0-b566-c409ebe83398" (UID: "2d9e6c18-e65d-42e0-b566-c409ebe83398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:30:21.224194 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.224164 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5" (OuterVolumeSpecName: "kube-api-access-wq6l5") pod "2d9e6c18-e65d-42e0-b566-c409ebe83398" (UID: "2d9e6c18-e65d-42e0-b566-c409ebe83398"). InnerVolumeSpecName "kube-api-access-wq6l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:21.322871 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.322834 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6l5\" (UniqueName: \"kubernetes.io/projected/2d9e6c18-e65d-42e0-b566-c409ebe83398-kube-api-access-wq6l5\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:21.322871 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.322863 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e6c18-e65d-42e0-b566-c409ebe83398-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:21.573299 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.573264 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d9e6c18-e65d-42e0-b566-c409ebe83398" containerID="56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1" exitCode=0 Jul 30 04:30:21.573541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.573316 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" Jul 30 04:30:21.573541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.573354 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" event={"ID":"2d9e6c18-e65d-42e0-b566-c409ebe83398","Type":"ContainerDied","Data":"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1"} Jul 30 04:30:21.573541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.573410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm" event={"ID":"2d9e6c18-e65d-42e0-b566-c409ebe83398","Type":"ContainerDied","Data":"810ddcff58ee210e925cff53009747c10e8eabc90f158274aa0b425b765ef3c2"} Jul 30 04:30:21.573541 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.573437 2568 scope.go:117] "RemoveContainer" containerID="56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1" Jul 30 04:30:21.583381 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.583361 2568 scope.go:117] "RemoveContainer" containerID="56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1" Jul 30 04:30:21.583678 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:21.583658 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1\": container with ID starting with 56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1 not found: ID does not exist" containerID="56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1" Jul 30 04:30:21.583752 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.583692 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1"} err="failed to get container status \"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1\": rpc error: code = NotFound desc = could not find container \"56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1\": container with ID starting with 56360485ba5b44de8ce7e9cdaa81d9d6ed0e586fb8f0b6bfa985df137600cbf1 not found: ID does not exist" Jul 30 04:30:21.595283 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.595257 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:30:21.597166 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:21.597146 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6c99-699774f8dd-9x6cm"] Jul 30 04:30:22.568278 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:22.568246 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9e6c18-e65d-42e0-b566-c409ebe83398" path="/var/lib/kubelet/pods/2d9e6c18-e65d-42e0-b566-c409ebe83398/volumes" Jul 30 04:30:27.199841 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:27.199803 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:30:27.200241 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:27.200069 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" containerID="cri-o://37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39" gracePeriod=30 Jul 30 04:30:32.451313 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.451287 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:30:32.529692 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529599 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsppc\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529692 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529684 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529722 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529744 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529781 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529827 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529862 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.529905 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.529897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert\") pod \"f529d16e-0992-47e7-9944-c207fb4b20cc\" (UID: \"f529d16e-0992-47e7-9944-c207fb4b20cc\") " Jul 30 04:30:32.530263 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530136 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:32.530322 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530302 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.530375 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530324 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:32.530559 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530425 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:30:32.530656 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530541 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:32.530880 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.530857 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:32.532213 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.532181 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:30:32.532320 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.532284 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc" (OuterVolumeSpecName: "kube-api-access-fsppc") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "kube-api-access-fsppc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:32.532560 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.532529 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:32.532674 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.532622 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f529d16e-0992-47e7-9944-c207fb4b20cc" (UID: "f529d16e-0992-47e7-9944-c207fb4b20cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:32.631236 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631201 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631236 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631236 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f529d16e-0992-47e7-9944-c207fb4b20cc-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631236 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631251 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsppc\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-kube-api-access-fsppc\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631264 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631277 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631308 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631331 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631346 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f529d16e-0992-47e7-9944-c207fb4b20cc-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:32.631816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631698 2568 generic.go:356] "Generic (PLEG): container finished" podID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerID="37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39" exitCode=0 Jul 30 04:30:32.631816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631799 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" Jul 30 04:30:32.631816 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631808 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerDied","Data":"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39"} Jul 30 04:30:32.631972 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631867 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8" event={"ID":"f529d16e-0992-47e7-9944-c207fb4b20cc","Type":"ContainerDied","Data":"c219e8e020a681a04b23723d3756b808f83632181823aba52be89a5387205b64"} Jul 30 04:30:32.631972 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.631915 2568 scope.go:117] "RemoveContainer" containerID="37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39" Jul 30 04:30:32.641288 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.641265 2568 scope.go:117] "RemoveContainer" containerID="86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44" Jul 30 04:30:32.651834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.651807 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:30:32.652811 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.652796 2568 scope.go:117] "RemoveContainer" containerID="37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39" Jul 30 04:30:32.653086 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:32.653070 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39\": container with ID starting with 37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39 not found: ID does not exist" containerID="37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39" Jul 30 04:30:32.653129 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.653095 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39"} err="failed to get container status \"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39\": rpc error: code = NotFound desc = could not find container \"37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39\": container with ID starting with 37b899e2eec98613f5a0b45ca9e4c6d4bacaee4908a4ff6999069f9fa369ee39 not found: ID does not exist" Jul 30 04:30:32.653129 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.653112 2568 scope.go:117] "RemoveContainer" containerID="86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44" Jul 30 04:30:32.653366 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:32.653349 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44\": container with ID starting with 86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44 not found: ID does not exist" containerID="86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44" Jul 30 04:30:32.653416 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.653369 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44"} err="failed to get container status \"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44\": rpc error: code = NotFound desc = could not find container \"86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44\": container with ID starting with 86798d5a017ff03c3881d58670e4786945483742778ae758d5da0cb2a30a0d44 not found: ID does not exist" Jul 30 04:30:32.654201 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:32.654182 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eb6c99-openshift-default-6bf564b5b8-fhjz8"] Jul 30 04:30:34.575143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:34.575093 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" path="/var/lib/kubelet/pods/f529d16e-0992-47e7-9944-c207fb4b20cc/volumes" Jul 30 04:30:43.200334 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.199923 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:30:43.200334 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.200225 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" podUID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" containerName="payload-processing" containerID="cri-o://876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327" gracePeriod=30 Jul 30 04:30:43.461044 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.460975 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:30:43.539637 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.539601 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl\") pod \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " Jul 30 04:30:43.539809 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.539699 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume\") pod \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\" (UID: \"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79\") " Jul 30 04:30:43.540118 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.540084 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" (UID: "386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:30:43.541847 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.541824 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl" (OuterVolumeSpecName: "kube-api-access-8sqzl") pod "386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" (UID: "386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79"). InnerVolumeSpecName "kube-api-access-8sqzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:43.641184 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.641146 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqzl\" (UniqueName: \"kubernetes.io/projected/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-kube-api-access-8sqzl\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:43.641184 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.641177 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:43.695005 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.694973 2568 generic.go:356] "Generic (PLEG): container finished" podID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" containerID="876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327" exitCode=0 Jul 30 04:30:43.695185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.695023 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" Jul 30 04:30:43.695185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.695026 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" event={"ID":"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79","Type":"ContainerDied","Data":"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327"} Jul 30 04:30:43.695185 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.695155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c" event={"ID":"386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79","Type":"ContainerDied","Data":"7ff4eda8e45d6237f93bfc12ab2a80279c07bf4bc46cebd587f28cab2486ca1e"} Jul 30 04:30:43.695296 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.695185 2568 scope.go:117] "RemoveContainer" containerID="876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327" Jul 30 04:30:43.705076 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.705058 2568 scope.go:117] "RemoveContainer" containerID="876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327" Jul 30 04:30:43.705319 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:43.705301 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327\": container with ID starting with 876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327 not found: ID does not exist" containerID="876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327" Jul 30 04:30:43.705363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.705331 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327"} err="failed to get container status \"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327\": rpc error: code = NotFound desc = could not find container \"876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327\": container with ID starting with 876f582bcdf93fa08e68cc0d4ff6d6b22aa04f915c33870c226efffe437ab327 not found: ID does not exist" Jul 30 04:30:43.716769 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.716694 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:30:43.720143 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:43.720122 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587da0-6fb4799cfb-vt77c"] Jul 30 04:30:44.567932 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:44.567885 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" path="/var/lib/kubelet/pods/386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79/volumes" Jul 30 04:30:49.726222 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:49.726190 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:30:49.726735 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:49.726389 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" containerID="cri-o://73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe" gracePeriod=30 Jul 30 04:30:50.601083 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601037 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:30:50.601523 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601506 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:50.601523 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601524 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:50.601635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601546 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9e6c18-e65d-42e0-b566-c409ebe83398" containerName="payload-pre-processing" Jul 30 04:30:50.601635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601551 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9e6c18-e65d-42e0-b566-c409ebe83398" containerName="payload-pre-processing" Jul 30 04:30:50.601635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601560 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" containerName="payload-processing" Jul 30 04:30:50.601635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601566 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" containerName="payload-processing" Jul 30 04:30:50.601635 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601632 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:50.601784 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601643 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="386dfdd3-99d2-4d5a-8d0c-2fb3ff02ee79" containerName="payload-processing" Jul 30 04:30:50.601784 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.601649 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9e6c18-e65d-42e0-b566-c409ebe83398" containerName="payload-pre-processing" Jul 30 04:30:50.606254 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.606234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.608801 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.608778 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-324a29-openshift-default-dockercfg-fh99c\"" Jul 30 04:30:50.616384 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.616358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:30:50.715254 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715254 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715551 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715551 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715609 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.715908 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.715720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbg66\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817317 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817317 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbg66\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817626 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817784 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.817918 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.818355 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.817995 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.818355 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.818172 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.818355 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.818226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.818507 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.818376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.820404 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.820380 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.820643 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.820624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.824951 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.824924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbg66\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.825151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.825133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token\") pod \"e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:50.918686 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:50.918651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:51.060649 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.060618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:30:51.063141 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:30:51.063103 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3adfd7_2476_4777_8b1d_d9a5848cb37a.slice/crio-773efcb4f03fcf5d03cf8e5fbe81a0be2ee23c80b14a844476dcf71bf9de695f WatchSource:0}: Error finding container 773efcb4f03fcf5d03cf8e5fbe81a0be2ee23c80b14a844476dcf71bf9de695f: Status 404 returned error can't find the container with id 773efcb4f03fcf5d03cf8e5fbe81a0be2ee23c80b14a844476dcf71bf9de695f Jul 30 04:30:51.065336 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.065303 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:30:51.065436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.065378 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:30:51.065436 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.065420 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:30:51.751421 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.751380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerStarted","Data":"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841"} Jul 30 04:30:51.751658 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.751456 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerStarted","Data":"773efcb4f03fcf5d03cf8e5fbe81a0be2ee23c80b14a844476dcf71bf9de695f"} Jul 30 04:30:51.771106 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.771055 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podStartSLOduration=1.7710385199999998 podStartE2EDuration="1.77103852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:51.769997795 +0000 UTC m=+2731.815496992" watchObservedRunningTime="2026-07-30 04:30:51.77103852 +0000 UTC m=+2731.816537716" Jul 30 04:30:51.919180 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:51.919146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:52.924707 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:52.924674 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:53.762340 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:53.762309 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:53.763207 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:53.763190 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:30:54.981579 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:54.981555 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:30:55.059892 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059802 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.059892 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059843 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.059892 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059892 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059913 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059934 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.059966 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4bpq\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060000 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060090 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060172 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060120 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket\") pod \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\" (UID: \"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0\") " Jul 30 04:30:55.060501 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060195 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:55.060501 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060381 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:55.060612 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060496 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:55.060670 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060607 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.060670 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060630 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.060670 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060647 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.060670 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060655 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:30:55.060973 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.060951 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:55.062578 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.062548 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq" (OuterVolumeSpecName: "kube-api-access-x4bpq") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "kube-api-access-x4bpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:55.062834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.062813 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:30:55.062834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.062818 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:30:55.062975 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.062895 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" (UID: "a23a77da-ec89-4b61-8a0e-4f18ec7e17d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:30:55.161686 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161649 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.161686 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161674 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.161686 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161683 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.161686 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161691 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bpq\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-kube-api-access-x4bpq\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.161950 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161699 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.161950 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.161708 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:30:55.428211 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428181 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:30:55.428633 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428619 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428695 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428635 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428695 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428644 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428695 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428649 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428695 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428670 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:55.428695 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428676 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:55.428858 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428741 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428858 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428749 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerName="istio-proxy" Jul 30 04:30:55.428858 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.428759 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f529d16e-0992-47e7-9944-c207fb4b20cc" containerName="istio-proxy" Jul 30 04:30:55.432194 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.432178 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.435486 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.435447 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-324a29\"" Jul 30 04:30:55.435637 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.435612 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-324a29-dockercfg-vdw8p\"" Jul 30 04:30:55.448046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.448021 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:30:55.464446 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.464419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.464660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.464601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzs7\" (UniqueName: \"kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.477998 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.477969 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:30:55.481524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.481506 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.491327 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.491304 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:30:55.566007 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.565960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzs7\" (UniqueName: \"kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.566007 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.566008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bvs\" (UniqueName: \"kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.566291 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.566083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.566291 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.566185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.567070 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.567045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.574048 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.574019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzs7\" (UniqueName: \"kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7\") pod \"payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.667350 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.667300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.667581 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.667455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bvs\" (UniqueName: \"kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.668065 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.668044 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.675346 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.675320 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bvs\" (UniqueName: \"kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs\") pod \"payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.742985 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.742901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:55.786922 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.786888 2568 generic.go:356] "Generic (PLEG): container finished" podID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" containerID="73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe" exitCode=0 Jul 30 04:30:55.787108 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.786967 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" Jul 30 04:30:55.787108 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.786981 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerDied","Data":"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe"} Jul 30 04:30:55.787108 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.787078 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf" event={"ID":"a23a77da-ec89-4b61-8a0e-4f18ec7e17d0","Type":"ContainerDied","Data":"af07e07761d1b13c0dde9bd6cf8f6d52eed34981199023adc97ca4ba2f63590c"} Jul 30 04:30:55.787262 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.787109 2568 scope.go:117] "RemoveContainer" containerID="73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe" Jul 30 04:30:55.792045 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.792024 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:55.799066 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.799048 2568 scope.go:117] "RemoveContainer" containerID="5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae" Jul 30 04:30:55.816630 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.815049 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:30:55.819843 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.819072 2568 scope.go:117] "RemoveContainer" containerID="73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe" Jul 30 04:30:55.820394 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:55.820071 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe\": container with ID starting with 73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe not found: ID does not exist" containerID="73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe" Jul 30 04:30:55.820394 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.820111 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe"} err="failed to get container status \"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe\": rpc error: code = NotFound desc = could not find container \"73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe\": container with ID starting with 73f82dd821d2886c69d61c5f62d734d9d501d57f12f0fcd780c49d82f4858bbe not found: ID does not exist" Jul 30 04:30:55.820394 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.820136 2568 scope.go:117] "RemoveContainer" containerID="5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae" Jul 30 04:30:55.821086 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:30:55.820992 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae\": container with ID starting with 5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae not found: ID does not exist" containerID="5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae" Jul 30 04:30:55.821086 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.821023 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae"} err="failed to get container status \"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae\": rpc error: code = NotFound desc = could not find container \"5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae\": container with ID starting with 5937df4a3a46bdfa6d082314db77dce684eca524a4a232a59729e44182cfecae not found: ID does not exist" Jul 30 04:30:55.821582 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.821562 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-587da0-openshift-default-785cf9fcb-hzncf"] Jul 30 04:30:55.902906 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.902868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:30:55.906546 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:30:55.905055 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaa3bbc_c697_4dcd_847d_8ad6c6a8ae58.slice/crio-649687c91640c02cd74ad080be669e15b36dcb183a755f2eceacba801e9c1802 WatchSource:0}: Error finding container 649687c91640c02cd74ad080be669e15b36dcb183a755f2eceacba801e9c1802: Status 404 returned error can't find the container with id 649687c91640c02cd74ad080be669e15b36dcb183a755f2eceacba801e9c1802 Jul 30 04:30:55.965448 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:55.965332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:30:55.968186 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:30:55.968160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca3e164_3625_4111_afa3_bd407f1f7366.slice/crio-4bbcaac3b9ced0f7ee6b59c9084df3da6f46732222e4382c34e5028cea046273 WatchSource:0}: Error finding container 4bbcaac3b9ced0f7ee6b59c9084df3da6f46732222e4382c34e5028cea046273: Status 404 returned error can't find the container with id 4bbcaac3b9ced0f7ee6b59c9084df3da6f46732222e4382c34e5028cea046273 Jul 30 04:30:56.569254 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.569218 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23a77da-ec89-4b61-8a0e-4f18ec7e17d0" path="/var/lib/kubelet/pods/a23a77da-ec89-4b61-8a0e-4f18ec7e17d0/volumes" Jul 30 04:30:56.794367 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.794335 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" event={"ID":"0ca3e164-3625-4111-afa3-bd407f1f7366","Type":"ContainerStarted","Data":"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2"} Jul 30 04:30:56.794601 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.794386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" event={"ID":"0ca3e164-3625-4111-afa3-bd407f1f7366","Type":"ContainerStarted","Data":"4bbcaac3b9ced0f7ee6b59c9084df3da6f46732222e4382c34e5028cea046273"} Jul 30 04:30:56.794601 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.794431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:30:56.797180 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.797150 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" event={"ID":"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58","Type":"ContainerStarted","Data":"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb"} Jul 30 04:30:56.797332 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.797191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" event={"ID":"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58","Type":"ContainerStarted","Data":"649687c91640c02cd74ad080be669e15b36dcb183a755f2eceacba801e9c1802"} Jul 30 04:30:56.797332 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.797216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:30:56.809702 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.809653 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" podStartSLOduration=1.809636695 podStartE2EDuration="1.809636695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:56.809196811 +0000 UTC m=+2736.854696006" watchObservedRunningTime="2026-07-30 04:30:56.809636695 +0000 UTC m=+2736.855135890" Jul 30 04:30:56.824796 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:30:56.824690 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" podStartSLOduration=1.824674623 podStartE2EDuration="1.824674623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:56.822134948 +0000 UTC m=+2736.867634143" watchObservedRunningTime="2026-07-30 04:30:56.824674623 +0000 UTC m=+2736.870173862" Jul 30 04:31:06.853131 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.853100 2568 generic.go:356] "Generic (PLEG): container finished" podID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerID="8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841" exitCode=0 Jul 30 04:31:06.853633 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.853178 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerDied","Data":"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841"} Jul 30 04:31:06.853633 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.853631 2568 scope.go:117] "RemoveContainer" containerID="8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841" Jul 30 04:31:06.854705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.854680 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:06.854774 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.854745 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:06.854812 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:06.854777 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:07.805917 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.805884 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:31:07.806116 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.806044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:31:07.878564 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.878521 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerStarted","Data":"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701"} Jul 30 04:31:07.919000 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.918967 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:31:07.920194 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.920170 2568 patch_prober.go:28] interesting pod/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 30 04:31:07.920253 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:07.920229 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 30 04:31:08.762677 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:08.762646 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:31:08.919693 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:08.919655 2568 patch_prober.go:28] interesting pod/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 30 04:31:08.920081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:08.919708 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 30 04:31:09.919009 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:09.918974 2568 patch_prober.go:28] interesting pod/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 30 04:31:09.919190 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:09.919044 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 30 04:31:10.919707 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:10.919670 2568 patch_prober.go:28] interesting pod/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 30 04:31:10.920108 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:10.919730 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 30 04:31:11.923069 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:11.923035 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:31:11.924080 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:11.924063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:31:19.314323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.314289 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:31:19.317048 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.317028 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.319876 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.319695 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-64af9f-openshift-default-dockercfg-5gpfm\"" Jul 30 04:31:19.330935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.330910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:31:19.406750 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.406750 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcz5\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.406969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407191 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.407027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.407191 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.407064 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508594 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508772 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508772 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508772 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcz5\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508772 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.508986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.508892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.509198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.509088 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.509198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.509133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.509198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.509175 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.509375 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.509359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.509619 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.509599 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.511303 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.511276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.511457 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.511437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.517247 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.517227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.517549 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.517531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcz5\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5\") pod \"e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.628282 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.628176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:19.775296 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.775271 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:31:19.777433 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:31:19.777404 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded166104_0c53_40f7_a12e_6f963c548522.slice/crio-37ecf968762ed322faaf15bff27936e219a19cc686348634aecaa78163e1c920 WatchSource:0}: Error finding container 37ecf968762ed322faaf15bff27936e219a19cc686348634aecaa78163e1c920: Status 404 returned error can't find the container with id 37ecf968762ed322faaf15bff27936e219a19cc686348634aecaa78163e1c920 Jul 30 04:31:19.779869 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.779836 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:19.779959 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.779904 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:19.779959 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.779931 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:19.945629 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.945595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerStarted","Data":"c48be20e75a3cce981c51574079bbb9d780182c557a844dc551369c3198730ae"} Jul 30 04:31:19.945813 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.945655 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerStarted","Data":"37ecf968762ed322faaf15bff27936e219a19cc686348634aecaa78163e1c920"} Jul 30 04:31:19.964691 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:19.964635 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podStartSLOduration=0.964614515 podStartE2EDuration="964.614515ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:31:19.963763755 +0000 UTC m=+2760.009262961" watchObservedRunningTime="2026-07-30 04:31:19.964614515 +0000 UTC m=+2760.010113709" Jul 30 04:31:20.628754 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:20.628721 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:20.633808 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:20.633787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:20.951215 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:20.951185 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:20.952155 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:20.952135 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:36.614067 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:36.614033 2568 patch_prober.go:28] interesting pod/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:37854->10.134.0.66:15021: read: connection reset by peer" start-of-body= Jul 30 04:31:36.614533 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:36.614120 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:37854->10.134.0.66:15021: read: connection reset by peer" Jul 30 04:31:37.042176 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.042137 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed166104-0c53-40f7-a12e-6f963c548522" containerID="c48be20e75a3cce981c51574079bbb9d780182c557a844dc551369c3198730ae" exitCode=0 Jul 30 04:31:37.042347 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.042210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerDied","Data":"c48be20e75a3cce981c51574079bbb9d780182c557a844dc551369c3198730ae"} Jul 30 04:31:37.042650 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.042636 2568 scope.go:117] "RemoveContainer" containerID="c48be20e75a3cce981c51574079bbb9d780182c557a844dc551369c3198730ae" Jul 30 04:31:37.043756 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.043729 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:37.043826 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.043793 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:37.043864 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:37.043825 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:31:38.051438 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:38.051398 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerStarted","Data":"777162f518fd915a50ae5b4ef772fc1b44e99e4d4574e46ffce66dc4b70feb36"} Jul 30 04:31:38.628721 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:38.628678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:38.628942 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:38.628918 2568 patch_prober.go:28] interesting pod/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 30 04:31:38.628990 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:38.628970 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 30 04:31:39.628761 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:39.628722 2568 patch_prober.go:28] interesting pod/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 30 04:31:39.629226 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:39.628781 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 30 04:31:40.629237 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:40.629160 2568 patch_prober.go:28] interesting pod/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 30 04:31:40.629237 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:40.629217 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 30 04:31:41.629657 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:41.629621 2568 patch_prober.go:28] interesting pod/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 30 04:31:41.630047 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:41.629692 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 30 04:31:42.631999 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:42.631967 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:42.632481 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:42.632440 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:31:42.633020 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:31:42.633000 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:35:20.912130 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:35:20.912097 2568 scope.go:117] "RemoveContainer" containerID="e3e579a1896c4cc58d32b5083ee189d86655ee28cda2fc3775d39deea2c46567" Jul 30 04:35:20.920963 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:35:20.920940 2568 scope.go:117] "RemoveContainer" containerID="1c2c8fcd29a5a97f0d2e1a91b3dc28422f9845a593491d3f92a404caa6709488" Jul 30 04:43:10.242942 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.242905 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:43:10.246672 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.246646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.249954 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.249911 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a1808a-openshift-default-dockercfg-csg9q\"" Jul 30 04:43:10.258526 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.258502 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:43:10.326322 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326843 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326843 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326708 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9sv\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.326843 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.326741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.427822 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.427777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.427822 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.427826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.427955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428092 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428210 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9sv\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428309 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428302 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428604 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428400 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428665 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.428870 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.428847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.430458 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.430436 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.430709 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.430691 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.438458 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.438424 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.439418 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.439390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9sv\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv\") pod \"e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.558836 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.558746 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:10.706793 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.706769 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:43:10.709183 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:43:10.709153 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487f416_6958_4c71_9c7d_f97c58df3b1e.slice/crio-3980a2d6acd6d12bd8ec88b3f133bea51884ed0760e804283496d3769454f8bc WatchSource:0}: Error finding container 3980a2d6acd6d12bd8ec88b3f133bea51884ed0760e804283496d3769454f8bc: Status 404 returned error can't find the container with id 3980a2d6acd6d12bd8ec88b3f133bea51884ed0760e804283496d3769454f8bc Jul 30 04:43:10.711640 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.711596 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:10.711748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.711671 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:10.711748 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.711699 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:10.737563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:10.737533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerStarted","Data":"3980a2d6acd6d12bd8ec88b3f133bea51884ed0760e804283496d3769454f8bc"} Jul 30 04:43:11.744570 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:11.744529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerStarted","Data":"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0"} Jul 30 04:43:11.763508 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:11.763443 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podStartSLOduration=1.7634245 podStartE2EDuration="1.7634245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:43:11.762073153 +0000 UTC m=+3471.807572348" watchObservedRunningTime="2026-07-30 04:43:11.7634245 +0000 UTC m=+3471.808923698" Jul 30 04:43:12.559684 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:12.559651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:12.568094 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:12.568067 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:12.750393 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:12.750358 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:12.751409 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:12.751390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:28.791710 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:28.791667 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 04:43:28.792099 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:28.791733 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 04:43:29.842021 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.841914 2568 generic.go:356] "Generic (PLEG): container finished" podID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerID="80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0" exitCode=0 Jul 30 04:43:29.842021 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.841985 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerDied","Data":"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0"} Jul 30 04:43:29.842516 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.842408 2568 scope.go:117] "RemoveContainer" containerID="80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0" Jul 30 04:43:29.843569 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.843541 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:29.843660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.843612 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:29.843660 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:29.843647 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 30 04:43:30.868490 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:30.868371 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerStarted","Data":"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116"} Jul 30 04:43:31.560257 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:31.560220 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:31.560683 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:31.560587 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 30 04:43:31.560872 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:31.560847 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 30 04:43:32.559588 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:32.559547 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 30 04:43:32.560920 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:32.560886 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 30 04:43:33.559497 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:33.559435 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 30 04:43:33.559835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:33.559804 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 30 04:43:34.560010 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:34.559968 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 30 04:43:34.560658 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:34.560626 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 30 04:43:35.563689 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:35.563659 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:35.564096 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:35.563936 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:43:35.564666 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:43:35.564648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:44:17.370020 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:44:17.369986 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:45:00.148568 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.148528 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp"] Jul 30 04:45:00.152443 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.152421 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch"] Jul 30 04:45:00.152570 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.152555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" Jul 30 04:45:00.155144 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.155114 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:45:00.155949 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.155928 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" Jul 30 04:45:00.161257 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.161226 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp"] Jul 30 04:45:00.165199 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.165173 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch"] Jul 30 04:45:00.171017 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.170994 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" Jul 30 04:45:00.178986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.178962 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" Jul 30 04:45:00.355437 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.355411 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch"] Jul 30 04:45:00.357539 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:00.357508 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4c56d_47d0_46ae_b095_dd7e778c7079.slice/crio-74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d WatchSource:0}: Error finding container 74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d: Status 404 returned error can't find the container with id 74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d Jul 30 04:45:00.534001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:00.533971 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp"] Jul 30 04:45:00.535782 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:00.535740 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a9c964_5b0d_4e7a_b1e6_b61a89832613.slice/crio-2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119 WatchSource:0}: Error finding container 2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119: Status 404 returned error can't find the container with id 2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119 Jul 30 04:45:01.350969 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.350930 2568 generic.go:356] "Generic (PLEG): container finished" podID="d3d4c56d-47d0-46ae-b095-dd7e778c7079" containerID="88ba82e11ff9cade74989844e10a35d6a678b95d436db7d7783a65faa6b0ab7a" exitCode=0 Jul 30 04:45:01.351431 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.351011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" event={"ID":"d3d4c56d-47d0-46ae-b095-dd7e778c7079","Type":"ContainerDied","Data":"88ba82e11ff9cade74989844e10a35d6a678b95d436db7d7783a65faa6b0ab7a"} Jul 30 04:45:01.351431 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.351049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" event={"ID":"d3d4c56d-47d0-46ae-b095-dd7e778c7079","Type":"ContainerStarted","Data":"74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d"} Jul 30 04:45:01.352739 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.352713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" event={"ID":"13a9c964-5b0d-4e7a-b1e6-b61a89832613","Type":"ContainerStarted","Data":"9edd4613bd24388abf686bd8cbc48f489858f08c720c60ca79a62c7fcdc05a4f"} Jul 30 04:45:01.352851 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.352759 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" event={"ID":"13a9c964-5b0d-4e7a-b1e6-b61a89832613","Type":"ContainerStarted","Data":"2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119"} Jul 30 04:45:01.377526 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:01.377450 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" podStartSLOduration=1.3774282279999999 podStartE2EDuration="1.377428228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:01.376846298 +0000 UTC m=+3581.422345493" watchObservedRunningTime="2026-07-30 04:45:01.377428228 +0000 UTC m=+3581.422927424" Jul 30 04:45:02.361089 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:02.361055 2568 generic.go:356] "Generic (PLEG): container finished" podID="13a9c964-5b0d-4e7a-b1e6-b61a89832613" containerID="9edd4613bd24388abf686bd8cbc48f489858f08c720c60ca79a62c7fcdc05a4f" exitCode=0 Jul 30 04:45:02.361606 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:02.361149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" event={"ID":"13a9c964-5b0d-4e7a-b1e6-b61a89832613","Type":"ContainerDied","Data":"9edd4613bd24388abf686bd8cbc48f489858f08c720c60ca79a62c7fcdc05a4f"} Jul 30 04:45:02.502688 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:02.502654 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" Jul 30 04:45:03.368385 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:03.368348 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" Jul 30 04:45:03.368868 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:03.368394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch" event={"ID":"d3d4c56d-47d0-46ae-b095-dd7e778c7079","Type":"ContainerDied","Data":"74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d"} Jul 30 04:45:03.368868 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:03.368507 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f75d247f707c5788ac5f78b4e3e293f6ad4169077ed4067fe09f806941452d" Jul 30 04:45:03.510139 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:03.510111 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" Jul 30 04:45:04.375974 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:04.375940 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" Jul 30 04:45:04.375974 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:04.375967 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756445-5lkfp" event={"ID":"13a9c964-5b0d-4e7a-b1e6-b61a89832613","Type":"ContainerDied","Data":"2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119"} Jul 30 04:45:04.376418 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:04.376009 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff61482855aee18a3631e2c725e371da0f1fc655544fccd07c7109e0966c119" Jul 30 04:45:04.569445 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:04.569411 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl"] Jul 30 04:45:04.569445 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:04.569449 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-l8gnl"] Jul 30 04:45:06.568523 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:06.568458 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5ceb01-9140-4025-be03-0ad8568e95e8" path="/var/lib/kubelet/pods/6e5ceb01-9140-4025-be03-0ad8568e95e8/volumes" Jul 30 04:45:09.597101 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:09.597065 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:45:14.632568 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.632482 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:14.633134 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633114 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d4c56d-47d0-46ae-b095-dd7e778c7079" containerName="cleanup" Jul 30 04:45:14.633224 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633137 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d4c56d-47d0-46ae-b095-dd7e778c7079" containerName="cleanup" Jul 30 04:45:14.633224 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633187 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a9c964-5b0d-4e7a-b1e6-b61a89832613" containerName="cleanup" Jul 30 04:45:14.633224 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633196 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a9c964-5b0d-4e7a-b1e6-b61a89832613" containerName="cleanup" Jul 30 04:45:14.633381 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633291 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a9c964-5b0d-4e7a-b1e6-b61a89832613" containerName="cleanup" Jul 30 04:45:14.633381 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.633306 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d4c56d-47d0-46ae-b095-dd7e778c7079" containerName="cleanup" Jul 30 04:45:14.637618 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.637599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.640002 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.639980 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1ddcb8\"" Jul 30 04:45:14.645133 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.645112 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:14.676048 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.676001 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:14.681907 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.681882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.684189 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.684160 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1ddcb8-dockercfg-r2z2j\"" Jul 30 04:45:14.687810 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.687786 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:14.710929 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.710901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dvc\" (UniqueName: \"kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.711072 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.710957 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.711072 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.711052 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6s4\" (UniqueName: \"kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.711151 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.711098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.812329 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.812279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dvc\" (UniqueName: \"kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.812558 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.812374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.812558 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.812451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6s4\" (UniqueName: \"kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.812558 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.812528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.813237 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.813210 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.813426 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.813403 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.821024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.820991 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6s4\" (UniqueName: \"kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4\") pod \"payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:14.821123 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.821084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dvc\" (UniqueName: \"kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.950223 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.950195 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:14.993735 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:14.993700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:15.110611 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.110579 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:15.111631 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:15.111597 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646e90fc_8c49_4ff9_b1b9_8c7c0a8a0c33.slice/crio-c5e88590ddae7367ed63f49b7e42c2f28c59e6a14bb5feebdda5aaca5656a864 WatchSource:0}: Error finding container c5e88590ddae7367ed63f49b7e42c2f28c59e6a14bb5feebdda5aaca5656a864: Status 404 returned error can't find the container with id c5e88590ddae7367ed63f49b7e42c2f28c59e6a14bb5feebdda5aaca5656a864 Jul 30 04:45:15.155504 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.155457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:15.157681 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:15.157655 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d518a2_fada_4727_a970_270cdeed626b.slice/crio-ef8793023dc6584fc98b4b092166c084bfe0707d43c726e8a1bdcca3e7374784 WatchSource:0}: Error finding container ef8793023dc6584fc98b4b092166c084bfe0707d43c726e8a1bdcca3e7374784: Status 404 returned error can't find the container with id ef8793023dc6584fc98b4b092166c084bfe0707d43c726e8a1bdcca3e7374784 Jul 30 04:45:15.439929 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.439892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" event={"ID":"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33","Type":"ContainerStarted","Data":"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb"} Jul 30 04:45:15.440140 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.439944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" event={"ID":"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33","Type":"ContainerStarted","Data":"c5e88590ddae7367ed63f49b7e42c2f28c59e6a14bb5feebdda5aaca5656a864"} Jul 30 04:45:15.440140 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.440071 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:15.441414 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.441389 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" event={"ID":"35d518a2-fada-4727-a970-270cdeed626b","Type":"ContainerStarted","Data":"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622"} Jul 30 04:45:15.441555 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.441433 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" event={"ID":"35d518a2-fada-4727-a970-270cdeed626b","Type":"ContainerStarted","Data":"ef8793023dc6584fc98b4b092166c084bfe0707d43c726e8a1bdcca3e7374784"} Jul 30 04:45:15.441555 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.441482 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:15.457297 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.457237 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" podStartSLOduration=1.45721595 podStartE2EDuration="1.45721595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:15.453904429 +0000 UTC m=+3595.499403632" watchObservedRunningTime="2026-07-30 04:45:15.45721595 +0000 UTC m=+3595.502715147" Jul 30 04:45:15.468542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:15.468414 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" podStartSLOduration=1.468394699 podStartE2EDuration="1.468394699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:15.467889105 +0000 UTC m=+3595.513388299" watchObservedRunningTime="2026-07-30 04:45:15.468394699 +0000 UTC m=+3595.513893895" Jul 30 04:45:21.026557 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:21.026528 2568 scope.go:117] "RemoveContainer" containerID="156afb065eb958f96843ad06b67c69dd99b0364daceb1734be0d064d69d2b615" Jul 30 04:45:26.449138 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:26.449101 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:26.449606 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:26.449389 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:38.595566 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.595531 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9"] Jul 30 04:45:38.598992 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.598973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:38.601032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.601010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:45:38.601146 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.601050 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:45:38.606909 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.606446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9"] Jul 30 04:45:38.662292 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.662249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9\" (UID: \"f3963531-ee15-44a1-88ef-6b746cfc2fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:38.762997 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.762961 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9\" (UID: \"f3963531-ee15-44a1-88ef-6b746cfc2fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:38.763682 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.763658 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9\" (UID: \"f3963531-ee15-44a1-88ef-6b746cfc2fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:38.910755 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:38.910670 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:39.050879 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:39.050854 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9"] Jul 30 04:45:39.053070 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:39.053016 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3963531_ee15_44a1_88ef_6b746cfc2fba.slice/crio-49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3 WatchSource:0}: Error finding container 49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3: Status 404 returned error can't find the container with id 49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3 Jul 30 04:45:39.585341 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:39.585242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" event={"ID":"f3963531-ee15-44a1-88ef-6b746cfc2fba","Type":"ContainerStarted","Data":"219c93773279c265210b36a70d5fd0f194fd1dd7a27fd4b40a342e28b8e6e157"} Jul 30 04:45:39.585341 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:39.585303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" event={"ID":"f3963531-ee15-44a1-88ef-6b746cfc2fba","Type":"ContainerStarted","Data":"49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3"} Jul 30 04:45:39.600696 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:39.600638 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" podStartSLOduration=1.600620326 podStartE2EDuration="1.600620326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:39.598260332 +0000 UTC m=+3619.643759527" watchObservedRunningTime="2026-07-30 04:45:39.600620326 +0000 UTC m=+3619.646119538" Jul 30 04:45:40.592000 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:40.591967 2568 generic.go:356] "Generic (PLEG): container finished" podID="f3963531-ee15-44a1-88ef-6b746cfc2fba" containerID="219c93773279c265210b36a70d5fd0f194fd1dd7a27fd4b40a342e28b8e6e157" exitCode=0 Jul 30 04:45:40.592191 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:40.592054 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" event={"ID":"f3963531-ee15-44a1-88ef-6b746cfc2fba","Type":"ContainerDied","Data":"219c93773279c265210b36a70d5fd0f194fd1dd7a27fd4b40a342e28b8e6e157"} Jul 30 04:45:41.737688 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:41.737662 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:41.793904 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:41.793878 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca\") pod \"f3963531-ee15-44a1-88ef-6b746cfc2fba\" (UID: \"f3963531-ee15-44a1-88ef-6b746cfc2fba\") " Jul 30 04:45:41.794243 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:41.794223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3963531-ee15-44a1-88ef-6b746cfc2fba" (UID: "f3963531-ee15-44a1-88ef-6b746cfc2fba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:45:41.895865 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:41.895767 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3963531-ee15-44a1-88ef-6b746cfc2fba-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:42.603646 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:42.603613 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" Jul 30 04:45:42.603815 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:42.603639 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9" event={"ID":"f3963531-ee15-44a1-88ef-6b746cfc2fba","Type":"ContainerDied","Data":"49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3"} Jul 30 04:45:42.603815 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:42.603696 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49472cabec0816f8a054a2e7478010c0543bb792c1d9fe9e0da7d9312a80f1d3" Jul 30 04:45:44.603292 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.603257 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:44.603786 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.603531 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" podUID="35d518a2-fada-4727-a970-270cdeed626b" containerName="payload-processing" containerID="cri-o://3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622" gracePeriod=30 Jul 30 04:45:44.614503 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.614425 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:44.614956 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.614903 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" podUID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" containerName="payload-pre-processing" containerID="cri-o://ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb" gracePeriod=30 Jul 30 04:45:44.756591 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.756176 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9"] Jul 30 04:45:44.759384 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.759353 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1ddcb8-g6ns9"] Jul 30 04:45:44.896304 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.896275 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:44.930430 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.930407 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf6s4\" (UniqueName: \"kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4\") pod \"35d518a2-fada-4727-a970-270cdeed626b\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " Jul 30 04:45:44.930580 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.930568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume\") pod \"35d518a2-fada-4727-a970-270cdeed626b\" (UID: \"35d518a2-fada-4727-a970-270cdeed626b\") " Jul 30 04:45:44.930947 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.930923 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35d518a2-fada-4727-a970-270cdeed626b" (UID: "35d518a2-fada-4727-a970-270cdeed626b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:45:44.932678 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.932655 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4" (OuterVolumeSpecName: "kube-api-access-lf6s4") pod "35d518a2-fada-4727-a970-270cdeed626b" (UID: "35d518a2-fada-4727-a970-270cdeed626b"). InnerVolumeSpecName "kube-api-access-lf6s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:45:44.942049 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:44.942027 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:45.031088 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.031051 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume\") pod \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " Jul 30 04:45:45.031264 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.031118 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9dvc\" (UniqueName: \"kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc\") pod \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\" (UID: \"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33\") " Jul 30 04:45:45.031427 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.031416 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d518a2-fada-4727-a970-270cdeed626b-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:45.031491 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.031406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" (UID: "646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:45:45.031491 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.031430 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf6s4\" (UniqueName: \"kubernetes.io/projected/35d518a2-fada-4727-a970-270cdeed626b-kube-api-access-lf6s4\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:45.033338 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.033316 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc" (OuterVolumeSpecName: "kube-api-access-p9dvc") pod "646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" (UID: "646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33"). InnerVolumeSpecName "kube-api-access-p9dvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:45:45.132495 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.132395 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:45.132495 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.132424 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dvc\" (UniqueName: \"kubernetes.io/projected/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33-kube-api-access-p9dvc\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:45.627935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.627889 2568 generic.go:356] "Generic (PLEG): container finished" podID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" containerID="ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb" exitCode=0 Jul 30 04:45:45.628402 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.627993 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" Jul 30 04:45:45.628527 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.628014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" event={"ID":"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33","Type":"ContainerDied","Data":"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb"} Jul 30 04:45:45.628527 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.628483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv" event={"ID":"646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33","Type":"ContainerDied","Data":"c5e88590ddae7367ed63f49b7e42c2f28c59e6a14bb5feebdda5aaca5656a864"} Jul 30 04:45:45.628527 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.628517 2568 scope.go:117] "RemoveContainer" containerID="ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb" Jul 30 04:45:45.629861 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.629811 2568 generic.go:356] "Generic (PLEG): container finished" podID="35d518a2-fada-4727-a970-270cdeed626b" containerID="3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622" exitCode=0 Jul 30 04:45:45.629930 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.629859 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" Jul 30 04:45:45.629930 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.629891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" event={"ID":"35d518a2-fada-4727-a970-270cdeed626b","Type":"ContainerDied","Data":"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622"} Jul 30 04:45:45.630017 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.629934 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m" event={"ID":"35d518a2-fada-4727-a970-270cdeed626b","Type":"ContainerDied","Data":"ef8793023dc6584fc98b4b092166c084bfe0707d43c726e8a1bdcca3e7374784"} Jul 30 04:45:45.638002 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.637975 2568 scope.go:117] "RemoveContainer" containerID="ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb" Jul 30 04:45:45.638329 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:45:45.638305 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb\": container with ID starting with ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb not found: ID does not exist" containerID="ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb" Jul 30 04:45:45.638432 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.638339 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb"} err="failed to get container status \"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb\": rpc error: code = NotFound desc = could not find container \"ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb\": container with ID starting with ab79c8e8a32db25a0d457e529b468770d3cd5e5ad6770320edd1f52446afeafb not found: ID does not exist" Jul 30 04:45:45.638432 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.638362 2568 scope.go:117] "RemoveContainer" containerID="3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622" Jul 30 04:45:45.646851 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.646837 2568 scope.go:117] "RemoveContainer" containerID="3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622" Jul 30 04:45:45.647083 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:45:45.647065 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622\": container with ID starting with 3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622 not found: ID does not exist" containerID="3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622" Jul 30 04:45:45.647127 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.647089 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622"} err="failed to get container status \"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622\": rpc error: code = NotFound desc = could not find container \"3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622\": container with ID starting with 3237e00ef06ae7dcf1d92037b5e1692edf7747e39ca1cd8417fab0b2cb93f622 not found: ID does not exist" Jul 30 04:45:45.654837 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.654811 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:45.659112 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.659092 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ddcb8-7d5557d588-5s75m"] Jul 30 04:45:45.672364 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.672340 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:45.674429 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:45.674410 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ddcb8-7b9f5c46b8-75qcv"] Jul 30 04:45:46.573877 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:46.569805 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d518a2-fada-4727-a970-270cdeed626b" path="/var/lib/kubelet/pods/35d518a2-fada-4727-a970-270cdeed626b/volumes" Jul 30 04:45:46.573877 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:46.570404 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" path="/var/lib/kubelet/pods/646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33/volumes" Jul 30 04:45:46.573877 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:46.570915 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3963531-ee15-44a1-88ef-6b746cfc2fba" path="/var/lib/kubelet/pods/f3963531-ee15-44a1-88ef-6b746cfc2fba/volumes" Jul 30 04:45:54.917558 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.917512 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8"] Jul 30 04:45:54.918001 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.917988 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" containerName="payload-pre-processing" Jul 30 04:45:54.918046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918003 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" containerName="payload-pre-processing" Jul 30 04:45:54.918046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918025 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3963531-ee15-44a1-88ef-6b746cfc2fba" containerName="revoke-keys" Jul 30 04:45:54.918046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918035 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3963531-ee15-44a1-88ef-6b746cfc2fba" containerName="revoke-keys" Jul 30 04:45:54.918046 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918046 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d518a2-fada-4727-a970-270cdeed626b" containerName="payload-processing" Jul 30 04:45:54.918219 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918054 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d518a2-fada-4727-a970-270cdeed626b" containerName="payload-processing" Jul 30 04:45:54.918219 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918141 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d518a2-fada-4727-a970-270cdeed626b" containerName="payload-processing" Jul 30 04:45:54.918219 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918151 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="646e90fc-8c49-4ff9-b1b9-8c7c0a8a0c33" containerName="payload-pre-processing" Jul 30 04:45:54.918219 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.918161 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3963531-ee15-44a1-88ef-6b746cfc2fba" containerName="revoke-keys" Jul 30 04:45:54.923032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.923007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:54.925583 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.925552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\"" Jul 30 04:45:54.925723 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.925613 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 04:45:54.928153 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:54.928126 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8"] Jul 30 04:45:55.041634 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.041597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8\" (UID: \"71a4da2e-77a0-44d8-963c-e7fe4f351bf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:55.142908 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.142870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8\" (UID: \"71a4da2e-77a0-44d8-963c-e7fe4f351bf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:55.143542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.143523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8\" (UID: \"71a4da2e-77a0-44d8-963c-e7fe4f351bf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:55.235823 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.235733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:55.380662 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.380628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8"] Jul 30 04:45:55.380901 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:45:55.380879 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a4da2e_77a0_44d8_963c_e7fe4f351bf9.slice/crio-0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91 WatchSource:0}: Error finding container 0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91: Status 404 returned error can't find the container with id 0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91 Jul 30 04:45:55.689263 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.689228 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" event={"ID":"71a4da2e-77a0-44d8-963c-e7fe4f351bf9","Type":"ContainerStarted","Data":"3f42a2607c774b01fa2edcc1ea8df5727f6961152b7c61a7e360cdadded77cbd"} Jul 30 04:45:55.689410 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.689282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" event={"ID":"71a4da2e-77a0-44d8-963c-e7fe4f351bf9","Type":"ContainerStarted","Data":"0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91"} Jul 30 04:45:55.705673 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:55.705622 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" podStartSLOduration=1.70560759 podStartE2EDuration="1.70560759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:55.704445523 +0000 UTC m=+3635.749944721" watchObservedRunningTime="2026-07-30 04:45:55.70560759 +0000 UTC m=+3635.751106783" Jul 30 04:45:56.696227 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:56.696188 2568 generic.go:356] "Generic (PLEG): container finished" podID="71a4da2e-77a0-44d8-963c-e7fe4f351bf9" containerID="3f42a2607c774b01fa2edcc1ea8df5727f6961152b7c61a7e360cdadded77cbd" exitCode=0 Jul 30 04:45:56.696652 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:56.696277 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" event={"ID":"71a4da2e-77a0-44d8-963c-e7fe4f351bf9","Type":"ContainerDied","Data":"3f42a2607c774b01fa2edcc1ea8df5727f6961152b7c61a7e360cdadded77cbd"} Jul 30 04:45:57.843737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:57.843708 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:57.975924 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:57.975828 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca\") pod \"71a4da2e-77a0-44d8-963c-e7fe4f351bf9\" (UID: \"71a4da2e-77a0-44d8-963c-e7fe4f351bf9\") " Jul 30 04:45:57.976176 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:57.976154 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71a4da2e-77a0-44d8-963c-e7fe4f351bf9" (UID: "71a4da2e-77a0-44d8-963c-e7fe4f351bf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:45:58.076850 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:58.076811 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a4da2e-77a0-44d8-963c-e7fe4f351bf9-maas-api-service-ca\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:45:58.708414 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:58.708372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" event={"ID":"71a4da2e-77a0-44d8-963c-e7fe4f351bf9","Type":"ContainerDied","Data":"0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91"} Jul 30 04:45:58.708414 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:58.708407 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8" Jul 30 04:45:58.708651 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:45:58.708428 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0633ee4228105d4d446df85f5834e8230206e82293ea171ef5bc9a1f8d636a91" Jul 30 04:46:10.668414 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:46:10.668367 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8"] Jul 30 04:46:10.670502 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:46:10.670477 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1808a-bndb8"] Jul 30 04:46:12.568247 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:46:12.568209 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a4da2e-77a0-44d8-963c-e7fe4f351bf9" path="/var/lib/kubelet/pods/71a4da2e-77a0-44d8-963c-e7fe4f351bf9/volumes" Jul 30 04:46:58.694906 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:46:58.694872 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:46:58.695553 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:46:58.695097 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" containerID="cri-o://a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116" gracePeriod=30 Jul 30 04:47:03.960973 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:03.960900 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:47:04.059024 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.058991 2568 generic.go:356] "Generic (PLEG): container finished" podID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerID="a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116" exitCode=0 Jul 30 04:47:04.059198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.059042 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" Jul 30 04:47:04.059198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.059078 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerDied","Data":"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116"} Jul 30 04:47:04.059198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.059138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn" event={"ID":"5487f416-6958-4c71-9c7d-f97c58df3b1e","Type":"ContainerDied","Data":"3980a2d6acd6d12bd8ec88b3f133bea51884ed0760e804283496d3769454f8bc"} Jul 30 04:47:04.059198 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.059161 2568 scope.go:117] "RemoveContainer" containerID="a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116" Jul 30 04:47:04.068258 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068204 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068258 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068240 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068377 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068261 2568 scope.go:117] "RemoveContainer" containerID="80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0" Jul 30 04:47:04.068377 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068302 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068377 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068330 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068406 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068446 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068529 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068565 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx9sv\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs\") pod \"5487f416-6958-4c71-9c7d-f97c58df3b1e\" (UID: \"5487f416-6958-4c71-9c7d-f97c58df3b1e\") " Jul 30 04:47:04.068687 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.068645 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:04.069050 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.069003 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.069299 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.069213 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:04.069299 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.069229 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:47:04.069299 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.069274 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:04.069546 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.069328 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:04.071313 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.071288 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:47:04.071407 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.071312 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:04.071631 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.071611 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:04.071698 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.071627 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv" (OuterVolumeSpecName: "kube-api-access-sx9sv") pod "5487f416-6958-4c71-9c7d-f97c58df3b1e" (UID: "5487f416-6958-4c71-9c7d-f97c58df3b1e"). InnerVolumeSpecName "kube-api-access-sx9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:04.092097 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.092075 2568 scope.go:117] "RemoveContainer" containerID="a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116" Jul 30 04:47:04.092456 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:04.092435 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116\": container with ID starting with a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116 not found: ID does not exist" containerID="a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116" Jul 30 04:47:04.092563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.092483 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116"} err="failed to get container status \"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116\": rpc error: code = NotFound desc = could not find container \"a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116\": container with ID starting with a0286cb94ba7097991a5540cf79cc6ee34ab0743ba704680c32143960c9f3116 not found: ID does not exist" Jul 30 04:47:04.092563 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.092510 2568 scope.go:117] "RemoveContainer" containerID="80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0" Jul 30 04:47:04.092789 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:04.092772 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0\": container with ID starting with 80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0 not found: ID does not exist" containerID="80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0" Jul 30 04:47:04.092835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.092797 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0"} err="failed to get container status \"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0\": rpc error: code = NotFound desc = could not find container \"80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0\": container with ID starting with 80ce7f2b2309a965066f6cbee7d3f2f944781fd27fcd379a9f3f79bf39944bc0 not found: ID does not exist" Jul 30 04:47:04.170696 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170656 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170696 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170692 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487f416-6958-4c71-9c7d-f97c58df3b1e-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170696 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170702 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170712 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170721 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9sv\" (UniqueName: \"kubernetes.io/projected/5487f416-6958-4c71-9c7d-f97c58df3b1e-kube-api-access-sx9sv\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170730 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170738 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.170935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.170745 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487f416-6958-4c71-9c7d-f97c58df3b1e-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:04.382323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.382286 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:47:04.387955 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.387922 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1808a-openshift-default-6747786b4d-llrvn"] Jul 30 04:47:04.568087 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:04.568053 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" path="/var/lib/kubelet/pods/5487f416-6958-4c71-9c7d-f97c58df3b1e/volumes" Jul 30 04:47:26.969801 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:26.969758 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch"] Jul 30 04:47:26.973144 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:26.973114 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-324a29-29756445-frbch"] Jul 30 04:47:27.074536 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.074498 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:47:27.074759 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.074711 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" podUID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" containerName="payload-processing" containerID="cri-o://727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb" gracePeriod=30 Jul 30 04:47:27.082609 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.082575 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:47:27.082855 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.082825 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" podUID="0ca3e164-3625-4111-afa3-bd407f1f7366" containerName="payload-pre-processing" containerID="cri-o://27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2" gracePeriod=30 Jul 30 04:47:27.378528 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.378505 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:47:27.381534 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.381508 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:47:27.504345 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504305 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76bvs\" (UniqueName: \"kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs\") pod \"0ca3e164-3625-4111-afa3-bd407f1f7366\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " Jul 30 04:47:27.504345 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504351 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fzs7\" (UniqueName: \"kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7\") pod \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " Jul 30 04:47:27.504616 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume\") pod \"0ca3e164-3625-4111-afa3-bd407f1f7366\" (UID: \"0ca3e164-3625-4111-afa3-bd407f1f7366\") " Jul 30 04:47:27.504616 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504562 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume\") pod \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\" (UID: \"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58\") " Jul 30 04:47:27.504875 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504846 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ca3e164-3625-4111-afa3-bd407f1f7366" (UID: "0ca3e164-3625-4111-afa3-bd407f1f7366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:47:27.504963 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504878 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" (UID: "6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:47:27.505008 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.504983 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:27.505008 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.505002 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca3e164-3625-4111-afa3-bd407f1f7366-plugins-config-volume\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:27.506838 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.506818 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7" (OuterVolumeSpecName: "kube-api-access-4fzs7") pod "6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" (UID: "6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58"). InnerVolumeSpecName "kube-api-access-4fzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:27.506838 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.506825 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs" (OuterVolumeSpecName: "kube-api-access-76bvs") pod "0ca3e164-3625-4111-afa3-bd407f1f7366" (UID: "0ca3e164-3625-4111-afa3-bd407f1f7366"). InnerVolumeSpecName "kube-api-access-76bvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:27.606101 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.605997 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76bvs\" (UniqueName: \"kubernetes.io/projected/0ca3e164-3625-4111-afa3-bd407f1f7366-kube-api-access-76bvs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:27.606101 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:27.606028 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzs7\" (UniqueName: \"kubernetes.io/projected/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58-kube-api-access-4fzs7\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:28.193238 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.193201 2568 generic.go:356] "Generic (PLEG): container finished" podID="0ca3e164-3625-4111-afa3-bd407f1f7366" containerID="27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2" exitCode=0 Jul 30 04:47:28.193737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.193251 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" Jul 30 04:47:28.193737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.193293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" event={"ID":"0ca3e164-3625-4111-afa3-bd407f1f7366","Type":"ContainerDied","Data":"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2"} Jul 30 04:47:28.193737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.193360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg" event={"ID":"0ca3e164-3625-4111-afa3-bd407f1f7366","Type":"ContainerDied","Data":"4bbcaac3b9ced0f7ee6b59c9084df3da6f46732222e4382c34e5028cea046273"} Jul 30 04:47:28.193737 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.193396 2568 scope.go:117] "RemoveContainer" containerID="27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2" Jul 30 04:47:28.194770 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.194723 2568 generic.go:356] "Generic (PLEG): container finished" podID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" containerID="727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb" exitCode=0 Jul 30 04:47:28.194835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.194768 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" Jul 30 04:47:28.194835 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.194786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" event={"ID":"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58","Type":"ContainerDied","Data":"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb"} Jul 30 04:47:28.194935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.194843 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv" event={"ID":"6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58","Type":"ContainerDied","Data":"649687c91640c02cd74ad080be669e15b36dcb183a755f2eceacba801e9c1802"} Jul 30 04:47:28.207594 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.207568 2568 scope.go:117] "RemoveContainer" containerID="27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2" Jul 30 04:47:28.207998 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:28.207975 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2\": container with ID starting with 27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2 not found: ID does not exist" containerID="27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2" Jul 30 04:47:28.208100 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.208005 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2"} err="failed to get container status \"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2\": rpc error: code = NotFound desc = could not find container \"27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2\": container with ID starting with 27b97b81d776b4c82add95d427602115b67c657151c776fe134a29af891c95e2 not found: ID does not exist" Jul 30 04:47:28.208100 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.208023 2568 scope.go:117] "RemoveContainer" containerID="727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb" Jul 30 04:47:28.216832 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.216812 2568 scope.go:117] "RemoveContainer" containerID="727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb" Jul 30 04:47:28.217087 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:28.217071 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb\": container with ID starting with 727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb not found: ID does not exist" containerID="727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb" Jul 30 04:47:28.217144 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.217096 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb"} err="failed to get container status \"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb\": rpc error: code = NotFound desc = could not find container \"727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb\": container with ID starting with 727807aee356c2b07a8d8c0a1dfa6080a02c95e2a7020d4ed2ba992cd2e084cb not found: ID does not exist" Jul 30 04:47:28.223690 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.223661 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:47:28.225733 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.225711 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-324a29-864d455d94-n4dsv"] Jul 30 04:47:28.237346 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.237317 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:47:28.239228 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.239209 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-324a29-547d454d5b-9dzbg"] Jul 30 04:47:28.567999 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.567967 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca3e164-3625-4111-afa3-bd407f1f7366" path="/var/lib/kubelet/pods/0ca3e164-3625-4111-afa3-bd407f1f7366/volumes" Jul 30 04:47:28.568428 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.568410 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" path="/var/lib/kubelet/pods/6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58/volumes" Jul 30 04:47:28.568691 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:28.568678 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d4c56d-47d0-46ae-b095-dd7e778c7079" path="/var/lib/kubelet/pods/d3d4c56d-47d0-46ae-b095-dd7e778c7079/volumes" Jul 30 04:47:33.722127 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:33.722092 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:47:33.722784 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:33.722386 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" containerID="cri-o://1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701" gracePeriod=30 Jul 30 04:47:38.985794 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:38.985728 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:47:39.123318 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123318 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123386 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123407 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbg66\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123432 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123613 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123509 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123814 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123682 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123814 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123762 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.123814 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123763 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:39.123814 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123796 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:39.124028 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123818 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:47:39.124028 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123853 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy\") pod \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\" (UID: \"fd3adfd7-2476-4777-8b1d-d9a5848cb37a\") " Jul 30 04:47:39.124028 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.123997 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:39.124366 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.124264 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.124366 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.124291 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.124366 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.124304 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.124366 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.124318 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.124616 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.124431 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:39.125974 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.125946 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:39.125974 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.125957 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66" (OuterVolumeSpecName: "kube-api-access-vbg66") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "kube-api-access-vbg66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:39.126132 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.126017 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:47:39.126132 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.126092 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd3adfd7-2476-4777-8b1d-d9a5848cb37a" (UID: "fd3adfd7-2476-4777-8b1d-d9a5848cb37a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:39.225828 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.225791 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.225828 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.225821 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.225828 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.225832 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbg66\" (UniqueName: \"kubernetes.io/projected/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-kube-api-access-vbg66\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.226081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.225843 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.226081 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.225852 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3adfd7-2476-4777-8b1d-d9a5848cb37a-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:39.258045 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.257949 2568 generic.go:356] "Generic (PLEG): container finished" podID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerID="1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701" exitCode=0 Jul 30 04:47:39.258045 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.258000 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" Jul 30 04:47:39.258045 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.258016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerDied","Data":"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701"} Jul 30 04:47:39.258301 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.258098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv" event={"ID":"fd3adfd7-2476-4777-8b1d-d9a5848cb37a","Type":"ContainerDied","Data":"773efcb4f03fcf5d03cf8e5fbe81a0be2ee23c80b14a844476dcf71bf9de695f"} Jul 30 04:47:39.258301 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.258122 2568 scope.go:117] "RemoveContainer" containerID="1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701" Jul 30 04:47:39.268027 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.268008 2568 scope.go:117] "RemoveContainer" containerID="8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841" Jul 30 04:47:39.277030 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.276615 2568 scope.go:117] "RemoveContainer" containerID="1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701" Jul 30 04:47:39.277030 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:39.276912 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701\": container with ID starting with 1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701 not found: ID does not exist" containerID="1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701" Jul 30 04:47:39.277030 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.276939 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701"} err="failed to get container status \"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701\": rpc error: code = NotFound desc = could not find container \"1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701\": container with ID starting with 1f0dfa1f9e67c97fe08e23767a4cc1610b600b8882be915b6a97cb4257a62701 not found: ID does not exist" Jul 30 04:47:39.277030 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.276965 2568 scope.go:117] "RemoveContainer" containerID="8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841" Jul 30 04:47:39.277331 ip-10-0-134-114 kubenswrapper[2568]: E0730 04:47:39.277288 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841\": container with ID starting with 8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841 not found: ID does not exist" containerID="8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841" Jul 30 04:47:39.277373 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.277340 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841"} err="failed to get container status \"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841\": rpc error: code = NotFound desc = could not find container \"8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841\": container with ID starting with 8dfc79903092c253f1d5de4974d17e8e95fb2bea9600efac83ae9ca445cf9841 not found: ID does not exist" Jul 30 04:47:39.279049 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.279026 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:47:39.282323 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:39.282301 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-324a29-openshift-default-6f5c466496-6xzzv"] Jul 30 04:47:40.567577 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:40.567544 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" path="/var/lib/kubelet/pods/fd3adfd7-2476-4777-8b1d-d9a5848cb37a/volumes" Jul 30 04:47:51.140064 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:51.140030 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:47:51.140517 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:51.140254 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" containerID="cri-o://777162f518fd915a50ae5b4ef772fc1b44e99e4d4574e46ffce66dc4b70feb36" gracePeriod=30 Jul 30 04:47:56.363452 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.363416 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed166104-0c53-40f7-a12e-6f963c548522" containerID="777162f518fd915a50ae5b4ef772fc1b44e99e4d4574e46ffce66dc4b70feb36" exitCode=0 Jul 30 04:47:56.363834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.363501 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerDied","Data":"777162f518fd915a50ae5b4ef772fc1b44e99e4d4574e46ffce66dc4b70feb36"} Jul 30 04:47:56.363834 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.363616 2568 scope.go:117] "RemoveContainer" containerID="c48be20e75a3cce981c51574079bbb9d780182c557a844dc551369c3198730ae" Jul 30 04:47:56.411004 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.410933 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:47:56.501142 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501099 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501142 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501145 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501178 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501199 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501243 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501273 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxcz5\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501401 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501375 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501768 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501440 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:56.501768 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501497 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo\") pod \"ed166104-0c53-40f7-a12e-6f963c548522\" (UID: \"ed166104-0c53-40f7-a12e-6f963c548522\") " Jul 30 04:47:56.501768 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501583 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:56.501768 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501745 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:56.501929 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501788 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 04:47:56.501981 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501923 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed166104-0c53-40f7-a12e-6f963c548522-istiod-ca-cert\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.501981 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501943 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-certs\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.501981 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501957 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-credential-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.501981 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.501971 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-workload-socket\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.502339 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.502160 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:56.503970 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.503942 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 04:47:56.503970 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.503949 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 04:47:56.503970 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.503958 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:56.504158 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.503964 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5" (OuterVolumeSpecName: "kube-api-access-wxcz5") pod "ed166104-0c53-40f7-a12e-6f963c548522" (UID: "ed166104-0c53-40f7-a12e-6f963c548522"). InnerVolumeSpecName "kube-api-access-wxcz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 04:47:56.602935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.602897 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcz5\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-kube-api-access-wxcz5\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.602935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.602932 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed166104-0c53-40f7-a12e-6f963c548522-istio-token\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.602935 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.602944 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed166104-0c53-40f7-a12e-6f963c548522-istio-podinfo\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.603179 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.602955 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-data\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:56.603179 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:56.602966 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed166104-0c53-40f7-a12e-6f963c548522-istio-envoy\") on node \"ip-10-0-134-114.ec2.internal\" DevicePath \"\"" Jul 30 04:47:57.369867 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.369826 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" event={"ID":"ed166104-0c53-40f7-a12e-6f963c548522","Type":"ContainerDied","Data":"37ecf968762ed322faaf15bff27936e219a19cc686348634aecaa78163e1c920"} Jul 30 04:47:57.369867 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.369863 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr" Jul 30 04:47:57.370392 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.369904 2568 scope.go:117] "RemoveContainer" containerID="777162f518fd915a50ae5b4ef772fc1b44e99e4d4574e46ffce66dc4b70feb36" Jul 30 04:47:57.385334 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.385306 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:47:57.389018 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.388991 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-64af9f-openshift-default-544c876cc6-jr7zr"] Jul 30 04:47:57.447977 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:57.447945 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67944866c5-sjcwh_e8e998a4-7071-4ed5-b2b4-8cb8c16e916d/manager/0.log" Jul 30 04:47:58.045529 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:58.045500 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-tp7gq_8d49e557-b78d-4e43-8cc8-b1ff47c865f2/manager/0.log" Jul 30 04:47:58.052325 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:58.052299 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-tp7gq_8d49e557-b78d-4e43-8cc8-b1ff47c865f2/copy-manifests/0.log" Jul 30 04:47:58.535285 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:58.535253 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756415-9cdbv_efbbe69f-d093-4a95-b828-4010e6e0d884/cleanup/0.log" Jul 30 04:47:58.568442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:58.568397 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed166104-0c53-40f7-a12e-6f963c548522" path="/var/lib/kubelet/pods/ed166104-0c53-40f7-a12e-6f963c548522/volumes" Jul 30 04:47:58.757427 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:47:58.757402 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756445-5lkfp_13a9c964-5b0d-4e7a-b1e6-b61a89832613/cleanup/0.log" Jul 30 04:48:01.355765 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:01.355734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6_779c70c3-62de-4d87-b035-1ac37190fb17/istio-proxy/0.log" Jul 30 04:48:01.477703 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:01.477668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-m2nrj_2e455cab-2974-4983-b37b-3e50b43a1711/discovery/0.log" Jul 30 04:48:01.825543 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:01.825509 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/1.log" Jul 30 04:48:02.067047 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:02.067015 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:48:02.653939 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:02.653906 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9fpgz_b271df55-cf62-4b27-8540-0aef5074613a/main/0.log" Jul 30 04:48:02.771559 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:02.771529 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4f7n7_e2e7218c-ba1d-49c9-b0fb-dccfbc39ddcc/main/0.log" Jul 30 04:48:11.043761 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:11.043673 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ch58f_5b45e529-005c-4897-890c-6c53db7030d3/global-pull-secret-syncer/0.log" Jul 30 04:48:11.245976 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:11.245944 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-znthr_cb61c933-717d-436b-a002-23b4f6f8a88e/konnectivity-agent/0.log" Jul 30 04:48:11.268079 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:11.268041 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-114.ec2.internal_12140044dd1a0f62cf96e929cee8dd45/haproxy/0.log" Jul 30 04:48:17.558432 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.558399 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/alertmanager/0.log" Jul 30 04:48:17.591542 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.591517 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/config-reloader/0.log" Jul 30 04:48:17.612827 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.612804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/kube-rbac-proxy-web/0.log" Jul 30 04:48:17.633482 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.633415 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/kube-rbac-proxy/0.log" Jul 30 04:48:17.659050 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.658960 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/kube-rbac-proxy-metric/0.log" Jul 30 04:48:17.684095 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.684076 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/prom-label-proxy/0.log" Jul 30 04:48:17.707224 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.707200 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ffbea0b-ece0-4af3-bd33-5543d36007c4/init-config-reloader/0.log" Jul 30 04:48:17.770989 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.770959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-state-metrics/0.log" Jul 30 04:48:17.791420 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.791394 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-rbac-proxy-main/0.log" Jul 30 04:48:17.811559 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.811525 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6z88r_400d1899-9cd9-4c9a-96d8-1cff2e417f4e/kube-rbac-proxy-self/0.log" Jul 30 04:48:17.839394 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.839362 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-684dd5868f-xm6zc_7db0b0f8-28b1-49c5-9772-ece2703fc684/metrics-server/0.log" Jul 30 04:48:17.869149 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.869125 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-n5p2p_d75da48f-0d01-4a6f-96e5-fd01b7518337/monitoring-plugin/0.log" Jul 30 04:48:17.976638 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.976609 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/node-exporter/0.log" Jul 30 04:48:17.997350 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:17.997293 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/kube-rbac-proxy/0.log" Jul 30 04:48:18.018385 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.018362 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px6sb_230b0d8b-67b2-4e76-9fe9-858119ee618e/init-textfile/0.log" Jul 30 04:48:18.154644 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.154601 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/kube-rbac-proxy-main/0.log" Jul 30 04:48:18.181705 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.181682 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/kube-rbac-proxy-self/0.log" Jul 30 04:48:18.209476 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.209448 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4sjgj_c300a760-a61e-4454-a5f2-98463d5387fe/openshift-state-metrics/0.log" Jul 30 04:48:18.248707 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.248635 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/prometheus/0.log" Jul 30 04:48:18.271601 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.271568 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/config-reloader/0.log" Jul 30 04:48:18.300621 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.300596 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/thanos-sidecar/0.log" Jul 30 04:48:18.323126 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.323094 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/kube-rbac-proxy-web/0.log" Jul 30 04:48:18.344206 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.344182 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/kube-rbac-proxy/0.log" Jul 30 04:48:18.364986 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.364959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/kube-rbac-proxy-thanos/0.log" Jul 30 04:48:18.386810 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.386785 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe0c8641-b074-4a2e-8901-5bcc3da10403/init-config-reloader/0.log" Jul 30 04:48:18.419859 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.419823 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v47mr_5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2/prometheus-operator/0.log" Jul 30 04:48:18.439841 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.439811 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v47mr_5ea1e26c-7f9c-48d8-a6ba-276c4d2d02d2/kube-rbac-proxy/0.log" Jul 30 04:48:18.468605 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.468576 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-7lf4s_f4cf6d70-21b0-4319-b424-07e4f8390b7b/prometheus-operator-admission-webhook/0.log" Jul 30 04:48:18.509194 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.509110 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/telemeter-client/0.log" Jul 30 04:48:18.530070 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.530045 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/reload/0.log" Jul 30 04:48:18.553739 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.553702 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9768d7845-5f4xm_1e90afbe-e730-42f7-9f25-7d203667789e/kube-rbac-proxy/0.log" Jul 30 04:48:18.583955 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.583931 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/thanos-query/0.log" Jul 30 04:48:18.604340 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.604317 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-web/0.log" Jul 30 04:48:18.624692 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.624669 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy/0.log" Jul 30 04:48:18.649438 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.649413 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/prom-label-proxy/0.log" Jul 30 04:48:18.671810 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.671780 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-rules/0.log" Jul 30 04:48:18.701307 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:18.701280 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd7d888b8-5xgx4_dd9ece92-d257-49c6-be38-4147864a1b66/kube-rbac-proxy-metrics/0.log" Jul 30 04:48:21.022757 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.022719 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg"] Jul 30 04:48:21.023168 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023154 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca3e164-3625-4111-afa3-bd407f1f7366" containerName="payload-pre-processing" Jul 30 04:48:21.023214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023170 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca3e164-3625-4111-afa3-bd407f1f7366" containerName="payload-pre-processing" Jul 30 04:48:21.023214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023179 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" Jul 30 04:48:21.023214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023185 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" Jul 30 04:48:21.023214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023200 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" containerName="payload-processing" Jul 30 04:48:21.023214 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023208 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" containerName="payload-processing" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023217 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023226 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023236 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023242 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023256 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023261 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023271 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023276 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023283 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a4da2e-77a0-44d8-963c-e7fe4f351bf9" containerName="revoke-keys" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023287 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a4da2e-77a0-44d8-963c-e7fe4f351bf9" containerName="revoke-keys" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023354 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed166104-0c53-40f7-a12e-6f963c548522" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023362 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca3e164-3625-4111-afa3-bd407f1f7366" containerName="payload-pre-processing" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023368 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a4da2e-77a0-44d8-963c-e7fe4f351bf9" containerName="revoke-keys" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023377 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023386 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aaa3bbc-c697-4dcd-847d-8ad6c6a8ae58" containerName="payload-processing" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023394 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3adfd7-2476-4777-8b1d-d9a5848cb37a" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023400 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" Jul 30 04:48:21.023524 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.023405 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5487f416-6958-4c71-9c7d-f97c58df3b1e" containerName="istio-proxy" Jul 30 04:48:21.026831 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.026810 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.029420 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.029397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qdzjx\"/\"default-dockercfg-vdm2s\"" Jul 30 04:48:21.029538 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.029402 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdzjx\"/\"openshift-service-ca.crt\"" Jul 30 04:48:21.030397 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.030377 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdzjx\"/\"kube-root-ca.crt\"" Jul 30 04:48:21.035560 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.035536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg"] Jul 30 04:48:21.151270 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.151229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-proc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.151442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.151301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-lib-modules\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.151442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.151336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xccsc\" (UniqueName: \"kubernetes.io/projected/273c0642-0bce-4e61-81a0-ee320aed45fb-kube-api-access-xccsc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.151442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.151360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-sys\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.151442 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.151422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-podres\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.252900 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.252865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-proc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253068 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.252926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-lib-modules\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253068 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.252968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccsc\" (UniqueName: \"kubernetes.io/projected/273c0642-0bce-4e61-81a0-ee320aed45fb-kube-api-access-xccsc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253068 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.252994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-sys\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253068 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.253014 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-proc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253068 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.253024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-podres\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253302 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.253081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-sys\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253302 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.253124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-podres\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.253302 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.253143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/273c0642-0bce-4e61-81a0-ee320aed45fb-lib-modules\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.268127 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.268099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccsc\" (UniqueName: \"kubernetes.io/projected/273c0642-0bce-4e61-81a0-ee320aed45fb-kube-api-access-xccsc\") pod \"perf-node-gather-daemonset-q8tqg\" (UID: \"273c0642-0bce-4e61-81a0-ee320aed45fb\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.338663 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.338571 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:21.506607 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:21.506583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg"] Jul 30 04:48:21.508845 ip-10-0-134-114 kubenswrapper[2568]: W0730 04:48:21.508817 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod273c0642_0bce_4e61_81a0_ee320aed45fb.slice/crio-d4368a1905c6a47946f8cc458ebfbaf7d31f7d1ec2ba3064c97a736be115e28a WatchSource:0}: Error finding container d4368a1905c6a47946f8cc458ebfbaf7d31f7d1ec2ba3064c97a736be115e28a: Status 404 returned error can't find the container with id d4368a1905c6a47946f8cc458ebfbaf7d31f7d1ec2ba3064c97a736be115e28a Jul 30 04:48:22.504852 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:22.504815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" event={"ID":"273c0642-0bce-4e61-81a0-ee320aed45fb","Type":"ContainerStarted","Data":"fce6cd13f81962e1f97275e12ce6105e7fdf65b47b67e9179e4e8a47c34493df"} Jul 30 04:48:22.505223 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:22.504871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:22.505223 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:22.504889 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" event={"ID":"273c0642-0bce-4e61-81a0-ee320aed45fb","Type":"ContainerStarted","Data":"d4368a1905c6a47946f8cc458ebfbaf7d31f7d1ec2ba3064c97a736be115e28a"} Jul 30 04:48:22.528728 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:22.528671 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" podStartSLOduration=1.528649535 podStartE2EDuration="1.528649535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 04:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:48:22.528259209 +0000 UTC m=+3782.573758400" watchObservedRunningTime="2026-07-30 04:48:22.528649535 +0000 UTC m=+3782.574148726" Jul 30 04:48:23.221152 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:23.221121 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5pwp_c59794b4-1156-4a33-8e85-79e5c858079a/dns/0.log" Jul 30 04:48:23.250681 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:23.250648 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5pwp_c59794b4-1156-4a33-8e85-79e5c858079a/kube-rbac-proxy/0.log" Jul 30 04:48:23.288985 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:23.288954 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xrnvf_8a801077-44f7-4e42-a883-50d978b8909e/dns-node-resolver/0.log" Jul 30 04:48:24.002013 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:24.001974 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wn5lb_f1c50bcc-64b6-4f18-8187-748c1e69fb49/node-ca/0.log" Jul 30 04:48:24.889763 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:24.889730 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6_779c70c3-62de-4d87-b035-1ac37190fb17/istio-proxy/0.log" Jul 30 04:48:25.043904 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:25.043871 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-m2nrj_2e455cab-2974-4983-b37b-3e50b43a1711/discovery/0.log" Jul 30 04:48:25.120666 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:25.120636 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/0.log" Jul 30 04:48:25.146694 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:25.146607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/1.log" Jul 30 04:48:25.206583 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:25.206552 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:48:25.991325 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:25.991299 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d5kn_0d5ad207-eeaf-4bf2-b21d-0dd9b1467f90/serve-healthcheck-canary/0.log" Jul 30 04:48:26.573187 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:26.573152 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8mzj_f856dd4b-e520-4d1d-8704-7adfff2953c5/kube-rbac-proxy/0.log" Jul 30 04:48:26.594164 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:26.594132 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8mzj_f856dd4b-e520-4d1d-8704-7adfff2953c5/exporter/0.log" Jul 30 04:48:26.616117 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:26.616094 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8mzj_f856dd4b-e520-4d1d-8704-7adfff2953c5/extractor/0.log" Jul 30 04:48:28.521150 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:28.521121 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-q8tqg" Jul 30 04:48:29.152126 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:29.152088 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67944866c5-sjcwh_e8e998a4-7071-4ed5-b2b4-8cb8c16e916d/manager/0.log" Jul 30 04:48:29.469980 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:29.469943 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-tp7gq_8d49e557-b78d-4e43-8cc8-b1ff47c865f2/manager/0.log" Jul 30 04:48:29.494649 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:29.494623 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-tp7gq_8d49e557-b78d-4e43-8cc8-b1ff47c865f2/copy-manifests/0.log" Jul 30 04:48:30.586031 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.585936 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6_779c70c3-62de-4d87-b035-1ac37190fb17/istio-proxy/0.log" Jul 30 04:48:30.742548 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.742518 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-m2nrj_2e455cab-2974-4983-b37b-3e50b43a1711/discovery/0.log" Jul 30 04:48:30.787363 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.787334 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/0.log" Jul 30 04:48:30.819017 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.818990 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/1.log" Jul 30 04:48:30.860750 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.860669 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:48:30.867652 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:30.867629 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69d4f5b96d-ghlvv_b7e259d5-da3e-4b53-b2b9-8f6ac377fab4/manager/0.log" Jul 30 04:48:32.175859 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:32.175827 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gnpf6_779c70c3-62de-4d87-b035-1ac37190fb17/istio-proxy/0.log" Jul 30 04:48:32.326625 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:32.326532 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-m2nrj_2e455cab-2974-4983-b37b-3e50b43a1711/discovery/0.log" Jul 30 04:48:32.377362 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:32.377312 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/0.log" Jul 30 04:48:32.405627 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:32.405590 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2tqpr_808a4069-437d-42e3-8dfc-8f612f5813bf/istio-proxy/1.log" Jul 30 04:48:32.446762 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:32.446732 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-chpmj_77d3fddb-3d99-4314-a9c0-d0ea8fe860ef/payload-processing/0.log" Jul 30 04:48:37.098712 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.098677 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/kube-multus-additional-cni-plugins/0.log" Jul 30 04:48:37.120032 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.120000 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/egress-router-binary-copy/0.log" Jul 30 04:48:37.141453 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.141430 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/cni-plugins/0.log" Jul 30 04:48:37.162000 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.161918 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/bond-cni-plugin/0.log" Jul 30 04:48:37.182967 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.182936 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/routeoverride-cni/0.log" Jul 30 04:48:37.206711 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.206687 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/whereabouts-cni-bincopy/0.log" Jul 30 04:48:37.227427 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.227401 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vrb7_f9ed8dbc-9e70-479f-b56f-988991b5c082/whereabouts-cni/0.log" Jul 30 04:48:37.623110 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.623079 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g2xcl_de93d457-c24b-40d8-8c37-2918bb3c691e/kube-multus/0.log" Jul 30 04:48:37.644197 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.644170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8tb9g_2df5d138-642c-4381-b83b-6ab0e3520c87/network-metrics-daemon/0.log" Jul 30 04:48:37.662825 ip-10-0-134-114 kubenswrapper[2568]: I0730 04:48:37.662800 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8tb9g_2df5d138-642c-4381-b83b-6ab0e3520c87/kube-rbac-proxy/0.log"