Sep 14 04:46:28.935428 ip-10-0-132-238 systemd[1]: Starting Kubernetes Kubelet... Sep 14 04:46:29.351729 ip-10-0-132-238 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 04:46:29.351729 ip-10-0-132-238 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 04:46:29.351729 ip-10-0-132-238 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 04:46:29.351729 ip-10-0-132-238 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 04:46:29.351729 ip-10-0-132-238 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 04:46:29.353638 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.353551 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 04:46:29.358668 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358652 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.358668 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358668 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358671 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358677 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358681 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358684 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358687 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358690 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358693 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358695 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358698 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358701 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358704 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358706 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358709 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358711 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358714 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358716 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358719 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358723 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358725 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.358731 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358728 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358730 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358733 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358736 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358738 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358741 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358744 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358746 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358749 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358752 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358754 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358757 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358759 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358761 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358764 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358767 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358769 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358771 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358774 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358776 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.359300 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358779 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358783 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358787 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358789 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358792 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358794 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358797 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358799 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358801 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358804 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358807 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358810 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358812 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358814 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358819 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358822 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358824 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358827 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358829 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358832 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.359792 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358835 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358837 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358840 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358842 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358845 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358847 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358849 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358853 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358857 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358861 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358864 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358867 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358869 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358872 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358874 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358877 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358879 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358882 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358885 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.360280 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358887 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358890 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358892 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358895 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358898 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358901 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358903 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358907 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358909 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358912 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358915 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358917 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358919 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358922 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358924 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358927 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358929 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358931 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.358934 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359632 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.360775 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359639 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359641 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359644 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359647 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359650 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359653 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359655 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359658 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359660 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359663 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359665 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359667 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359670 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359673 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359675 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359677 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359680 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359683 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359686 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.361263 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359691 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359695 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359698 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359701 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359704 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359706 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359709 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359712 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359715 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359717 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359720 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359722 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359724 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359727 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359729 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359732 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359734 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359737 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359739 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.361726 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359742 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359744 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359747 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359750 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359753 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359755 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359758 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359762 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359766 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359769 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359772 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359774 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359778 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359781 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359784 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359786 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359789 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359792 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359795 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359798 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.362181 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359800 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359803 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359806 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359808 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359811 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359813 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359816 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359819 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359821 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359823 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359826 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359828 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359831 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359834 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359836 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359838 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359841 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359843 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359846 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359848 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.362672 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359851 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359854 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359856 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359858 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359862 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359865 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359867 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359870 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359872 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359875 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359877 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359879 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359882 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359886 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359889 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359892 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359895 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359897 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359900 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.363168 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.359903 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360319 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360325 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360328 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360331 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360334 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360339 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360342 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360344 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360346 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360349 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360352 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360355 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360357 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360359 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360363 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360367 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360369 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360373 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.363639 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360375 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360378 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360381 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360383 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360386 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360388 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360391 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360393 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360396 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360398 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360401 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360403 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360406 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360411 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360413 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360416 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360419 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360421 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360424 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360428 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360431 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.364097 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360433 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360436 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360438 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360441 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360443 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360446 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360448 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360451 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360453 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360456 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360458 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360461 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360466 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360469 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360472 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360474 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360477 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360479 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360482 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.364608 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360484 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360487 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360489 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360492 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360494 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360496 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360499 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360502 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360505 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360509 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360512 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360516 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360519 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360522 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360524 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360527 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360530 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360532 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360535 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.365060 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360538 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360540 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360542 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360545 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360547 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360550 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360553 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360555 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360557 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360560 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360562 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360565 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360568 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360571 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360573 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360575 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360578 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360581 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360583 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360585 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.365572 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360589 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.360592 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360658 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360693 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360700 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360704 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360708 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360711 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360715 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360722 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360725 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360728 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360732 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360735 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360738 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360741 2581 flags.go:64] FLAG: --cgroup-root="" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360744 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360747 2581 flags.go:64] FLAG: --client-ca-file="" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360750 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360753 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360758 2581 flags.go:64] FLAG: --cluster-domain="" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360760 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360764 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360767 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 04:46:29.366055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360770 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360774 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360777 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360780 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360783 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360786 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360789 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360792 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360797 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360800 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360804 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360806 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360811 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360814 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360817 2581 flags.go:64] FLAG: --enable-server="true" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360820 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360824 2581 flags.go:64] FLAG: --event-burst="100" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360827 2581 flags.go:64] FLAG: --event-qps="50" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360830 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360833 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360836 2581 flags.go:64] FLAG: --eviction-hard="" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360840 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360842 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360845 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360848 2581 flags.go:64] FLAG: --eviction-soft="" Sep 14 04:46:29.366653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360851 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360854 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360857 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360860 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360863 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360866 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360869 2581 flags.go:64] FLAG: --feature-gates="" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360872 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360875 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360878 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360881 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360884 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360887 2581 flags.go:64] FLAG: --help="false" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360890 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360893 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360901 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360904 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360907 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360910 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360915 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360918 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360920 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360923 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 04:46:29.367254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360926 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360929 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360932 2581 flags.go:64] FLAG: --kube-reserved="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360935 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360938 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360941 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360943 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360946 2581 flags.go:64] FLAG: --lock-file="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360949 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360952 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360955 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360960 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360963 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360966 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360969 2581 flags.go:64] FLAG: --logging-format="text" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360972 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360975 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360978 2581 flags.go:64] FLAG: --manifest-url="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360981 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360985 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360988 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360992 2581 flags.go:64] FLAG: --max-pods="110" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360994 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.360997 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361001 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 04:46:29.367839 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361004 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361007 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361010 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361014 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361021 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361024 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361027 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361030 2581 flags.go:64] FLAG: --pod-cidr="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361033 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361038 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361041 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361044 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361047 2581 flags.go:64] FLAG: --port="10250" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361050 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361053 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fb5211d99f6aac6f" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361056 2581 flags.go:64] FLAG: --qos-reserved="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361059 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361062 2581 flags.go:64] FLAG: --register-node="true" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361065 2581 flags.go:64] FLAG: --register-with-taints="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361069 2581 flags.go:64] FLAG: --registry-burst="10" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361072 2581 flags.go:64] FLAG: --registry-qps="5" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361075 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361078 2581 flags.go:64] FLAG: --reserved-memory="" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361081 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361084 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 04:46:29.368455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361087 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361090 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361092 2581 flags.go:64] FLAG: --runonce="false" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361096 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361099 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361102 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361106 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361109 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361112 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361128 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361131 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361135 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361138 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361141 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361144 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361147 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361150 2581 flags.go:64] FLAG: --system-cgroups="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361153 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361158 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361161 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361163 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361168 2581 flags.go:64] FLAG: --tls-min-version="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361170 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361173 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361176 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 04:46:29.369104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361179 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361182 2581 flags.go:64] FLAG: --v="2" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361186 2581 flags.go:64] FLAG: --version="false" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361190 2581 flags.go:64] FLAG: --vmodule="" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361194 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.361198 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361312 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361316 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361319 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361322 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361325 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361327 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361330 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361334 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361337 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361339 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361342 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361344 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361347 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361350 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361352 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.369728 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361355 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361358 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361361 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361363 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361366 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361368 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361371 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361373 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361376 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361378 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361381 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361384 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361386 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361388 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361391 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361393 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361397 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361399 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361402 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.370516 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361404 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361407 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361409 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361412 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361414 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361418 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361421 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361423 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361426 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361428 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361431 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361433 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361436 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361439 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361441 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361444 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361446 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361449 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361451 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361454 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.371011 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361456 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361459 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361461 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361464 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361466 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361469 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361471 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361474 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361477 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361479 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361482 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361485 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361487 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361490 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361492 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361495 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361497 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361501 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361505 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361508 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.371521 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361511 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361514 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361517 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361520 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361522 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361525 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361529 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361532 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361535 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361538 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361541 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361544 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361546 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361549 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361551 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361554 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361556 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361559 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361561 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361564 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.371999 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361566 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361570 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361573 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361576 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.361579 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.362304 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.371912 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.371930 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.371945 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.371952 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372007 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372011 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372014 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372018 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.372502 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372021 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372024 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372027 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372030 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372033 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372035 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372038 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372041 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372043 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372046 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372048 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372051 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372056 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372059 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372062 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372065 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372068 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372071 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372074 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.372863 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372076 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372080 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372083 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372086 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372088 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372091 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372094 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372097 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372099 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372102 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372104 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372107 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372110 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372112 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372128 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372131 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372134 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372136 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372139 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372141 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.373337 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372144 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372147 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372149 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372153 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372157 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372160 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372163 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372166 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372169 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372171 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372174 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372176 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372179 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372182 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372185 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372187 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372191 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372194 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372197 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.373811 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372199 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372202 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372204 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372207 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372210 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372212 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372215 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372217 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372220 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372223 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372226 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372228 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372231 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372233 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372236 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372238 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372241 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372244 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372246 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372249 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.374317 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372251 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372254 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372256 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372259 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372261 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372264 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372267 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372269 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372272 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372275 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372277 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372280 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372282 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372285 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372288 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372290 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.374802 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372293 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.372298 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372408 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372413 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372416 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372419 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372421 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372424 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372427 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372429 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372432 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372435 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372438 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372440 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372443 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 04:46:29.375210 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372446 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372448 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372451 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372453 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372456 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372458 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372461 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372464 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372466 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372469 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372472 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372474 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372477 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372479 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372482 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372485 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372487 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372490 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372492 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372495 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 04:46:29.375580 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372498 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372500 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372502 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372505 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372508 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372511 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372514 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372518 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372522 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372525 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372528 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372530 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372533 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372536 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372538 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372541 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372543 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372546 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372549 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 04:46:29.376047 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372552 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372554 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372557 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372559 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372562 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372564 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372567 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372569 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372572 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372574 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372577 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372579 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372582 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372584 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372588 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372591 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372594 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372597 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372600 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 04:46:29.376523 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372602 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372605 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372608 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372611 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372613 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372616 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372618 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372621 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372623 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372626 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372628 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372631 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372635 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372638 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372641 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372644 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372646 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372649 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372652 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 04:46:29.377005 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372654 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372657 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372659 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372662 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372664 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372667 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372669 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372672 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:29.372674 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.372679 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.372785 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.376210 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.377242 2581 server.go:1019] "Starting client certificate rotation" Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.377345 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 04:46:29.377594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.377383 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 04:46:29.380112 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.380096 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 04:46:29.386944 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.386926 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 04:46:29.399056 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.399036 2581 log.go:25] "Validated CRI v1 runtime API" Sep 14 04:46:29.404101 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.404078 2581 log.go:25] "Validated CRI v1 image API" Sep 14 04:46:29.405800 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.405781 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 04:46:29.408436 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.408413 2581 fs.go:135] Filesystem UUIDs: map[1be5d64d-251c-4c7b-8b23-332a19cbcf8b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7d5fb1a-6b25-45c3-99b2-f8c745ebdd4b:/dev/nvme0n1p4] Sep 14 04:46:29.408498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.408435 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 04:46:29.410206 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.410184 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 04:46:29.414362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.414038 2581 manager.go:216] Machine: {Timestamp:2026-09-14 04:46:29.413205072 +0000 UTC m=+0.370821800 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199829 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec213c2be27884484fe6a6bfc0662e88 SystemUUID:ec213c2b-e278-8448-4fe6-a6bfc0662e88 BootID:f344f2d1-0cee-43c4-951d-0e381dea01a3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:7f:f2:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:7f:f2:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d9:45:14:82:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 04:46:29.414362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.414354 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 04:46:29.414498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.414486 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 04:46:29.416513 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.416489 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 04:46:29.416646 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.416515 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 04:46:29.418053 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.418043 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 04:46:29.418090 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.418056 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 04:46:29.418090 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.418064 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 04:46:29.418900 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.418887 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 04:46:29.420109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.420099 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 04:46:29.424653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424642 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 14 04:46:29.424768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424759 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 04:46:29.424798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424788 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 14 04:46:29.424798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424797 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 04:46:29.424859 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424811 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 04:46:29.424859 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424819 2581 kubelet.go:398] "Adding apiserver pod source" Sep 14 04:46:29.424859 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.424828 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 04:46:29.425205 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.425194 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 04:46:29.425255 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.425215 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 04:46:29.427511 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.427496 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 04:46:29.430557 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.430543 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 04:46:29.430602 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.430574 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 04:46:29.432648 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432633 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432653 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432660 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432664 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432673 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432679 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432691 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432697 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432703 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 04:46:29.432708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432709 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 04:46:29.432938 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432724 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 04:46:29.432938 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.432730 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 04:46:29.433675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.433662 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 04:46:29.433744 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.433677 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 04:46:29.435084 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.435065 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 04:46:29.435328 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.435302 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 04:46:29.437531 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.437519 2581 server.go:1268] "Started kubelet" Sep 14 04:46:29.437713 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.437652 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 04:46:29.437814 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.437733 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 04:46:29.438460 ip-10-0-132-238 systemd[1]: Started Kubernetes Kubelet. Sep 14 04:46:29.438734 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.438705 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 04:46:29.439304 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.439283 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 04:46:29.439973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.439940 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 14 04:46:29.442786 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.442765 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 04:46:29.443640 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.443615 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 04:46:29.443970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.442822 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-238.ec2.internal.18d51685774ac609 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-238.ec2.internal,UID:ip-10-0-132-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-238.ec2.internal,},FirstTimestamp:2026-09-14 04:46:29.437490697 +0000 UTC m=+0.395107424,LastTimestamp:2026-09-14 04:46:29.437490697 +0000 UTC m=+0.395107424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-238.ec2.internal,}" Sep 14 04:46:29.446735 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.446718 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 04:46:29.447508 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.447445 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 04:46:29.447668 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.447647 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.448323 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448308 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 04:46:29.448323 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448323 2581 factory.go:55] Registering systemd factory Sep 14 04:46:29.448323 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448332 2581 factory.go:223] Registration of the systemd container factory successfully Sep 14 04:46:29.448512 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448450 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 04:46:29.448512 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448451 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 04:46:29.448512 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448487 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448559 2581 factory.go:153] Registering CRI-O factory Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448571 2581 factory.go:223] Registration of the crio container factory successfully Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448587 2581 factory.go:103] Registering Raw factory Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448596 2581 manager.go:1196] Started watching for new ooms in manager Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448602 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 14 04:46:29.448647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.448633 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 14 04:46:29.449279 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.449263 2581 manager.go:349] Starting recovery of all containers Sep 14 04:46:29.452260 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.452235 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 04:46:29.452386 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.452361 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 04:46:29.462586 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.462554 2581 manager.go:354] Recovery completed Sep 14 04:46:29.463827 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.463805 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gpmrd" Sep 14 04:46:29.466769 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.466757 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.468376 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.468360 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.468446 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.468388 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.468446 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.468398 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.469024 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469010 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 04:46:29.469024 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469022 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 04:46:29.469179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469036 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 14 04:46:29.469179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469065 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.469738 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469720 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.469810 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469750 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.469810 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.469760 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.470751 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.470677 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-238.ec2.internal.18d5168579220d56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-238.ec2.internal,UID:ip-10-0-132-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-238.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-238.ec2.internal,},FirstTimestamp:2026-09-14 04:46:29.468376406 +0000 UTC m=+0.425993135,LastTimestamp:2026-09-14 04:46:29.468376406 +0000 UTC m=+0.425993135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-238.ec2.internal,}" Sep 14 04:46:29.471177 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.471161 2581 policy_none.go:49] "None policy: Start" Sep 14 04:46:29.471177 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.471179 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 04:46:29.471301 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.471189 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 04:46:29.473458 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.473338 2581 policy_none.go:47] "Start" Sep 14 04:46:29.473542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.473512 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gpmrd" Sep 14 04:46:29.478318 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.478232 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-238.ec2.internal.18d5168579224e5f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-238.ec2.internal,UID:ip-10-0-132-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-238.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-238.ec2.internal,},FirstTimestamp:2026-09-14 04:46:29.468393055 +0000 UTC m=+0.426009781,LastTimestamp:2026-09-14 04:46:29.468393055 +0000 UTC m=+0.426009781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-238.ec2.internal,}" Sep 14 04:46:29.480303 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.480219 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-238.ec2.internal.18d5168579227086 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-238.ec2.internal,UID:ip-10-0-132-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-238.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-238.ec2.internal,},FirstTimestamp:2026-09-14 04:46:29.468401798 +0000 UTC m=+0.426018526,LastTimestamp:2026-09-14 04:46:29.468401798 +0000 UTC m=+0.426018526,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-238.ec2.internal,}" Sep 14 04:46:29.495487 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.495471 2581 manager.go:337] "Starting Device Plugin manager" Sep 14 04:46:29.495589 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.495511 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 04:46:29.495589 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.495521 2581 server.go:87] "Starting device plugin registration server" Sep 14 04:46:29.495825 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.495811 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 04:46:29.495879 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.495827 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 04:46:29.495959 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.495945 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 04:46:29.496098 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.496029 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 04:46:29.496098 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.496038 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 04:46:29.496663 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.496635 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 04:46:29.496732 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.496676 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.544647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.544615 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 04:46:29.546867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.545723 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 04:46:29.546867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.545769 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 04:46:29.546867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.545793 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 04:46:29.546867 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.545828 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 04:46:29.548600 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.548582 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:29.596740 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.596720 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.597691 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.597674 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.597770 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.597704 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.597770 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.597713 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.597770 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.597739 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.609139 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.609054 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.609139 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.609076 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-238.ec2.internal\": node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.624945 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.624925 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.646560 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.646529 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal"] Sep 14 04:46:29.646640 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.646627 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.647533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.647519 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.647609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.647544 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.647609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.647554 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.649727 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.649708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.649781 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.649736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.649910 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.649897 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.650078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.650112 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650094 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.650653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650638 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.650709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650655 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.650709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650666 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.650709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650676 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.650709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650677 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.650709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.650686 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.652946 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.652931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.653005 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.652954 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:29.653590 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.653576 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:29.653644 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.653605 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:29.653644 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.653619 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:29.679841 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.679822 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-238.ec2.internal\" not found" node="ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.683931 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.683915 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-238.ec2.internal\" not found" node="ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.725965 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.725946 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.750486 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.750462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.750597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.750491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.750597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.750517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/729843ab636a52a35ed9dc96e1e6fe1d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-238.ec2.internal\" (UID: \"729843ab636a52a35ed9dc96e1e6fe1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.750597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.750558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.750597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.750583 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a417062cfd8d99b4997d23d1a6f32503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal\" (UID: \"a417062cfd8d99b4997d23d1a6f32503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.827065 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.827037 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.850787 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.850767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/729843ab636a52a35ed9dc96e1e6fe1d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-238.ec2.internal\" (UID: \"729843ab636a52a35ed9dc96e1e6fe1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.850851 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.850801 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/729843ab636a52a35ed9dc96e1e6fe1d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-238.ec2.internal\" (UID: \"729843ab636a52a35ed9dc96e1e6fe1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.927577 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:29.927522 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:29.982988 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.982956 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:29.986947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:29.986924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:30.027890 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.027865 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.128466 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.128435 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.228935 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.228878 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.268292 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.268268 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:30.329985 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.329941 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.378047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.378008 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 04:46:30.378628 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.378271 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 04:46:30.378628 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.378272 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 04:46:30.431075 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.431040 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.480092 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.480015 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 04:41:30 +0000 UTC" deadline="2028-05-22 19:35:42.582316614 +0000 UTC" Sep 14 04:46:30.480092 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.480047 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14798h49m12.102273054s" Sep 14 04:46:30.532271 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.532236 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.570376 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:30.570331 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda417062cfd8d99b4997d23d1a6f32503.slice/crio-f7de36491dd0f6f9f18977fea342c68ada67a23bb6c8c19fe4d6709d73a13d50 WatchSource:0}: Error finding container f7de36491dd0f6f9f18977fea342c68ada67a23bb6c8c19fe4d6709d73a13d50: Status 404 returned error can't find the container with id f7de36491dd0f6f9f18977fea342c68ada67a23bb6c8c19fe4d6709d73a13d50 Sep 14 04:46:30.570875 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:30.570853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729843ab636a52a35ed9dc96e1e6fe1d.slice/crio-ad2f2fd56e0fe3ec02116206b7682d1cccf2e76fc4b25bfab3256482ae1e5427 WatchSource:0}: Error finding container ad2f2fd56e0fe3ec02116206b7682d1cccf2e76fc4b25bfab3256482ae1e5427: Status 404 returned error can't find the container with id ad2f2fd56e0fe3ec02116206b7682d1cccf2e76fc4b25bfab3256482ae1e5427 Sep 14 04:46:30.577524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.577504 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 04:46:30.607096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.607068 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 04:46:30.608030 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.608013 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientMemory" Sep 14 04:46:30.608095 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.608043 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 04:46:30.608095 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.608056 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeHasSufficientPID" Sep 14 04:46:30.632687 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.632665 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.708348 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.708322 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:30.733789 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.733729 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.836193 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.836166 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.937104 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:30.937081 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-238.ec2.internal\" not found" Sep 14 04:46:30.994205 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:30.994117 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:31.048179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.048151 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" Sep 14 04:46:31.058319 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.058303 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 04:46:31.059391 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.059378 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" Sep 14 04:46:31.073347 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.073331 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 04:46:31.426195 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.425833 2581 apiserver.go:52] "Watching apiserver" Sep 14 04:46:31.433015 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.432979 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 04:46:31.434702 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.434669 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-j8764","openshift-multus/network-metrics-daemon-xdm24","openshift-network-diagnostics/network-check-target-cbv9t","openshift-network-operator/iptables-alerter-wpvt7","openshift-ovn-kubernetes/ovnkube-node-9qsvh","kube-system/global-pull-secret-syncer-lw9b4","kube-system/konnectivity-agent-m7964","kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8","openshift-cluster-node-tuning-operator/tuned-spjtp","openshift-image-registry/node-ca-445p5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal","openshift-multus/multus-jtpdv","openshift-dns/node-resolver-pwbjw"] Sep 14 04:46:31.437497 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.437474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.441150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.441126 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.443895 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.443870 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 04:46:31.444147 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.444116 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 04:46:31.444370 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.444351 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cmpc\"" Sep 14 04:46:31.444792 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.444772 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.445013 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.444997 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:31.445090 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.445066 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:31.445157 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.445077 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xbbzg\"" Sep 14 04:46:31.445278 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.444997 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 04:46:31.445278 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.445275 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 04:46:31.445469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.445279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.445469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.445387 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.447211 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447183 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.447329 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447239 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.447329 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447191 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 04:46:31.447603 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.447678 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447642 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 04:46:31.447920 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.447592 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xch56\"" Sep 14 04:46:31.449905 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.449440 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.449905 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.449844 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-894jg\"" Sep 14 04:46:31.450729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.450698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.450823 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.450789 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:31.451530 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.451502 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 04:46:31.452397 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.452370 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.452542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.452522 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 04:46:31.452684 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.452664 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 04:46:31.452962 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.452924 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 04:46:31.453305 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.453286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.455641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.455620 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.456650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.456385 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7g4rv\"" Sep 14 04:46:31.456650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.456409 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 04:46:31.456921 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.456853 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.457345 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.457314 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.459241 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.459223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.459337 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.459315 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:31.459926 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.459892 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gltrn\"" Sep 14 04:46:31.460168 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.460147 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 04:46:31.460317 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.460157 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 04:46:31.461179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cnibin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461285 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-multus-certs\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461285 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64k8\" (UniqueName: \"kubernetes.io/projected/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-kube-api-access-s64k8\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.461285 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-systemd-units\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461285 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-netd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-env-overrides\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-kubelet-config\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-system-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-socket-dir-parent\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-daemon-config\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-etc-kubernetes\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-script-lib\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8kr\" (UniqueName: \"kubernetes.io/projected/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-kube-api-access-cd8kr\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cni-binary-copy\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-slash\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b332df2d-27ca-4bb8-baad-7aadb1148973-konnectivity-ca\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-multus\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-etc-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461724 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-netns\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-host-slash\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-node-log\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.461882 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-log-socket\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b332df2d-27ca-4bb8-baad-7aadb1148973-agent-certs\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-k8s-cni-cncf-io\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.461978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-bin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-hostroot\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462043 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-conf-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nsh\" (UniqueName: \"kubernetes.io/projected/a88d88d6-16f8-4645-9d28-1db5b39be0a9-kube-api-access-25nsh\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-iptables-alerter-script\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-kubelet\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-netns\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-systemd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-var-lib-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-bin\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-config\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.462533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.463362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-kubelet\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.463362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-ovn\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.463362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.462419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-dbus\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.463362 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.463164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.464364 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.464136 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 04:46:31.464712 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.464539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.466861 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.466591 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.466861 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.466709 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdzj5\"" Sep 14 04:46:31.467002 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.466906 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.467002 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.466956 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.467413 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.467387 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.467731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.467682 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.467941 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.467850 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 04:46:31.467941 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.467877 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2scrm\"" Sep 14 04:46:31.469736 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.469167 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 04:46:31.469736 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.469570 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rqxz\"" Sep 14 04:46:31.470002 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.469798 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 04:46:31.484123 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.484100 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hfhrc" Sep 14 04:46:31.494450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.494429 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hfhrc" Sep 14 04:46:31.550345 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.550313 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 04:46:31.553143 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.553092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" event={"ID":"a417062cfd8d99b4997d23d1a6f32503","Type":"ContainerStarted","Data":"f7de36491dd0f6f9f18977fea342c68ada67a23bb6c8c19fe4d6709d73a13d50"} Sep 14 04:46:31.554783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.554756 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" event={"ID":"729843ab636a52a35ed9dc96e1e6fe1d","Type":"ContainerStarted","Data":"ad2f2fd56e0fe3ec02116206b7682d1cccf2e76fc4b25bfab3256482ae1e5427"} Sep 14 04:46:31.562715 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cnibin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.562803 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64k8\" (UniqueName: \"kubernetes.io/projected/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-kube-api-access-s64k8\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.562803 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-systemd-units\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.562803 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-netd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-env-overrides\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-systemd\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-tuned\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-daemon-config\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-etc-kubernetes\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8kr\" (UniqueName: \"kubernetes.io/projected/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-kube-api-access-cd8kr\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgvq\" (UniqueName: \"kubernetes.io/projected/66f3737b-d741-4c5c-8788-f184f954777e-kube-api-access-2qgvq\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.562995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-conf\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-slash\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9dc\" (UniqueName: \"kubernetes.io/projected/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-kube-api-access-nf9dc\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b332df2d-27ca-4bb8-baad-7aadb1148973-konnectivity-ca\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3371066-7823-4f23-95db-b5d47a1ce426-host\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b10002bc-68a5-479a-a081-6e7f8a6cccb4-hosts-file\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysconfig\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-netns\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-node-log\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563361 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-log-socket\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3371066-7823-4f23-95db-b5d47a1ce426-serviceca\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-registration-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563492 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-sys\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563513 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-tmp\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-bin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.563609 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-hostroot\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.563606 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-env-overrides\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-conf-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-iptables-alerter-script\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564317 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cnibin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-netns\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-netns\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.564457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-config\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564558 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-modprobe-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-kubelet\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.564798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564741 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-systemd-units\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-cnibin\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-multus-certs\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.564997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjvf\" (UniqueName: \"kubernetes.io/projected/f3371066-7823-4f23-95db-b5d47a1ce426-kube-api-access-ggjvf\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6t2\" (UniqueName: \"kubernetes.io/projected/b10002bc-68a5-479a-a081-6e7f8a6cccb4-kube-api-access-lh6t2\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.565097 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-system-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565470 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565026 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-config\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.565470 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-netd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.565470 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565236 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-system-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565470 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-slash\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.565679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565514 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-etc-kubernetes\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565520 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-cni-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565869 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565832 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-kubelet\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.565954 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565935 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-multus-certs\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.566009 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565991 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.566063 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.565997 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b332df2d-27ca-4bb8-baad-7aadb1148973-konnectivity-ca\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.566119 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.566091 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:31.566175 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566112 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.566259 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566169 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-iptables-alerter-script\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.566259 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.566207 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:46:32.066145593 +0000 UTC m=+3.023762324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:31.566482 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.566583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-netns\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.566642 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566628 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-node-log\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.566703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566653 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 04:46:31.566703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-log-socket\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.566799 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566771 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-bin\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.566799 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566773 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-daemon-config\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.566902 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566834 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-hostroot\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.566902 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-conf-dir\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567007 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-socket-dir-parent\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567007 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.566984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-script-lib\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.567109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-var-lib-kubelet\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.567109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cni-binary-copy\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-multus\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567298 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-etc-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.567298 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-device-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.567298 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-sys-fs\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.567450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-multus-socket-dir-parent\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.567450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567427 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.567450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-etc-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.567649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-host-slash\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.567649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567575 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-var-lib-cni-multus\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.567649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567583 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-socket-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.567797 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.567797 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-host\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.567797 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567773 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-host-slash\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.567949 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b332df2d-27ca-4bb8-baad-7aadb1148973-agent-certs\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.567949 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-k8s-cni-cncf-io\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.568052 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.567981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nsh\" (UniqueName: \"kubernetes.io/projected/a88d88d6-16f8-4645-9d28-1db5b39be0a9-kube-api-access-25nsh\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.568052 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:31.568052 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-kubelet\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568224 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-systemd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568224 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-bin\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568433 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568412 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovnkube-script-lib\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568492 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-kubelet-config\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.568541 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-var-lib-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568599 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10002bc-68a5-479a-a081-6e7f8a6cccb4-tmp-dir\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.568599 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.568698 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwml\" (UniqueName: \"kubernetes.io/projected/10863011-efa7-4c2b-95eb-dba333bd1ee2-kube-api-access-llwml\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.568698 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.568698 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.568859 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-ovn\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.568909 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.568889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d88d6-16f8-4645-9d28-1db5b39be0a9-cni-binary-copy\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.569089 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-dbus\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.569168 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-kubernetes\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.569168 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-run\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.569291 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-lib-modules\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.569291 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnk8\" (UniqueName: \"kubernetes.io/projected/c054e08e-84af-4062-bd3c-4456cea1f2e5-kube-api-access-ghnk8\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.569392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-ovn\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.569392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-kubelet\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.569392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569362 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-host-cni-bin\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.569542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569398 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88d88d6-16f8-4645-9d28-1db5b39be0a9-host-run-k8s-cni-cncf-io\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.569542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-run-systemd\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.569542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569473 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-kubelet-config\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.569650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569542 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-var-lib-openvswitch\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.569761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.569740 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-dbus\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:31.574150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.574040 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b332df2d-27ca-4bb8-baad-7aadb1148973-agent-certs\") pod \"konnectivity-agent-m7964\" (UID: \"b332df2d-27ca-4bb8-baad-7aadb1148973\") " pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.574681 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.574657 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.578714 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.578651 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8kr\" (UniqueName: \"kubernetes.io/projected/6ba67880-ddb9-4e90-887e-fca7b4d93b0f-kube-api-access-cd8kr\") pod \"ovnkube-node-9qsvh\" (UID: \"6ba67880-ddb9-4e90-887e-fca7b4d93b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.580837 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.580792 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64k8\" (UniqueName: \"kubernetes.io/projected/bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340-kube-api-access-s64k8\") pod \"iptables-alerter-wpvt7\" (UID: \"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340\") " pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.581748 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.581375 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:31.581748 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.581397 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:31.581748 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.581410 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:31.581748 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.581470 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:32.081451666 +0000 UTC m=+3.039068397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:31.585804 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.585757 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nsh\" (UniqueName: \"kubernetes.io/projected/a88d88d6-16f8-4645-9d28-1db5b39be0a9-kube-api-access-25nsh\") pod \"multus-jtpdv\" (UID: \"a88d88d6-16f8-4645-9d28-1db5b39be0a9\") " pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.670008 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.669971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-modprobe-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670008 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-cnibin\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjvf\" (UniqueName: \"kubernetes.io/projected/f3371066-7823-4f23-95db-b5d47a1ce426-kube-api-access-ggjvf\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6t2\" (UniqueName: \"kubernetes.io/projected/b10002bc-68a5-479a-a081-6e7f8a6cccb4-kube-api-access-lh6t2\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670157 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.670163 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670216 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-modprobe-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-var-lib-kubelet\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-cnibin\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:31.670255 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:32.170234499 +0000 UTC m=+3.127851231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:31.670286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-var-lib-kubelet\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-device-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-sys-fs\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670417 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-device-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-socket-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670524 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-sys-fs\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670544 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-d\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-host\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10002bc-68a5-479a-a081-6e7f8a6cccb4-tmp-dir\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwml\" (UniqueName: \"kubernetes.io/projected/10863011-efa7-4c2b-95eb-dba333bd1ee2-kube-api-access-llwml\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670641 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-socket-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-host\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.670849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-kubernetes\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-run\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-lib-modules\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnk8\" (UniqueName: \"kubernetes.io/projected/c054e08e-84af-4062-bd3c-4456cea1f2e5-kube-api-access-ghnk8\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-systemd\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-kubernetes\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-tuned\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670963 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10002bc-68a5-479a-a081-6e7f8a6cccb4-tmp-dir\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgvq\" (UniqueName: \"kubernetes.io/projected/66f3737b-d741-4c5c-8788-f184f954777e-kube-api-access-2qgvq\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-conf\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9dc\" (UniqueName: \"kubernetes.io/projected/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-kube-api-access-nf9dc\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3371066-7823-4f23-95db-b5d47a1ce426-host\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671107 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-lib-modules\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b10002bc-68a5-479a-a081-6e7f8a6cccb4-hosts-file\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.671651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b10002bc-68a5-479a-a081-6e7f8a6cccb4-hosts-file\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671232 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3371066-7823-4f23-95db-b5d47a1ce426-host\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671246 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671249 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysconfig\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3371066-7823-4f23-95db-b5d47a1ce426-serviceca\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-systemd\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-registration-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671347 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-sys\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysctl-conf\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-tmp\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-sysconfig\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.670698 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10863011-efa7-4c2b-95eb-dba333bd1ee2-registration-dir\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f3737b-d741-4c5c-8788-f184f954777e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671474 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-run\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671501 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f3737b-d741-4c5c-8788-f184f954777e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.672354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671517 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-sys\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.672864 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.671853 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3371066-7823-4f23-95db-b5d47a1ce426-serviceca\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.673723 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.673702 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-tmp\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.673822 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.673768 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-etc-tuned\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.684058 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.683998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgvq\" (UniqueName: \"kubernetes.io/projected/66f3737b-d741-4c5c-8788-f184f954777e-kube-api-access-2qgvq\") pod \"multus-additional-cni-plugins-j8764\" (UID: \"66f3737b-d741-4c5c-8788-f184f954777e\") " pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.684520 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.684493 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6t2\" (UniqueName: \"kubernetes.io/projected/b10002bc-68a5-479a-a081-6e7f8a6cccb4-kube-api-access-lh6t2\") pod \"node-resolver-pwbjw\" (UID: \"b10002bc-68a5-479a-a081-6e7f8a6cccb4\") " pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:31.684787 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.684765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjvf\" (UniqueName: \"kubernetes.io/projected/f3371066-7823-4f23-95db-b5d47a1ce426-kube-api-access-ggjvf\") pod \"node-ca-445p5\" (UID: \"f3371066-7823-4f23-95db-b5d47a1ce426\") " pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.685632 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.685376 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwml\" (UniqueName: \"kubernetes.io/projected/10863011-efa7-4c2b-95eb-dba333bd1ee2-kube-api-access-llwml\") pod \"aws-ebs-csi-driver-node-fbrg8\" (UID: \"10863011-efa7-4c2b-95eb-dba333bd1ee2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.686935 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.686916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnk8\" (UniqueName: \"kubernetes.io/projected/c054e08e-84af-4062-bd3c-4456cea1f2e5-kube-api-access-ghnk8\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:31.687367 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.687349 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9dc\" (UniqueName: \"kubernetes.io/projected/6ae1c54a-ac2d-4f55-ad47-7ae478fd466e-kube-api-access-nf9dc\") pod \"tuned-spjtp\" (UID: \"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e\") " pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.755230 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.755198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:31.763716 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.763690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jtpdv" Sep 14 04:46:31.776666 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.776643 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpvt7" Sep 14 04:46:31.782506 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.782486 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:31.785246 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.785223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:31.792631 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.792610 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" Sep 14 04:46:31.801460 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.801441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8764" Sep 14 04:46:31.809350 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.809333 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spjtp" Sep 14 04:46:31.817199 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.817177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-445p5" Sep 14 04:46:31.824053 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:31.824030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwbjw" Sep 14 04:46:32.074732 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.074651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:32.074911 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.074814 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:32.074911 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.074893 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:46:33.074868437 +0000 UTC m=+4.032485160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:32.175835 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.175791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:32.176002 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.175866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:32.176002 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.175951 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:32.176002 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.175968 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:32.176002 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.175994 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:32.176002 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.176003 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:33.175989068 +0000 UTC m=+4.133605798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:32.176312 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.176008 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:32.176312 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.176052 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:33.176039122 +0000 UTC m=+4.133655860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:32.498466 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.498369 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 04:41:32 +0000 UTC" deadline="2028-06-26 17:24:07.909069892 +0000 UTC" Sep 14 04:46:32.498466 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.498405 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15636h37m35.410668179s" Sep 14 04:46:32.546776 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:32.546733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:32.546946 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:32.546896 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:32.738717 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.738640 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10002bc_68a5_479a_a081_6e7f8a6cccb4.slice/crio-c14fbe86e4fed22da65230c42505885d3f619ac5be7508e3380539f5648c9a0c WatchSource:0}: Error finding container c14fbe86e4fed22da65230c42505885d3f619ac5be7508e3380539f5648c9a0c: Status 404 returned error can't find the container with id c14fbe86e4fed22da65230c42505885d3f619ac5be7508e3380539f5648c9a0c Sep 14 04:46:32.739908 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.739870 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88d88d6_16f8_4645_9d28_1db5b39be0a9.slice/crio-09540b7ca215fe4ca1e9caf7c4a5b6277a1e805ee8a5ef80b4687308893fe936 WatchSource:0}: Error finding container 09540b7ca215fe4ca1e9caf7c4a5b6277a1e805ee8a5ef80b4687308893fe936: Status 404 returned error can't find the container with id 09540b7ca215fe4ca1e9caf7c4a5b6277a1e805ee8a5ef80b4687308893fe936 Sep 14 04:46:32.743857 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.743809 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3371066_7823_4f23_95db_b5d47a1ce426.slice/crio-6a10aff500668ed075b1f55bcfe12eb522c37c2be4f0db5447d54f68be5f31b6 WatchSource:0}: Error finding container 6a10aff500668ed075b1f55bcfe12eb522c37c2be4f0db5447d54f68be5f31b6: Status 404 returned error can't find the container with id 6a10aff500668ed075b1f55bcfe12eb522c37c2be4f0db5447d54f68be5f31b6 Sep 14 04:46:32.749191 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.749095 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10863011_efa7_4c2b_95eb_dba333bd1ee2.slice/crio-01cd4c833cf6d1e1fada5cbeb9ceaa7adf409abd731b230c217360bb3b13d5ce WatchSource:0}: Error finding container 01cd4c833cf6d1e1fada5cbeb9ceaa7adf409abd731b230c217360bb3b13d5ce: Status 404 returned error can't find the container with id 01cd4c833cf6d1e1fada5cbeb9ceaa7adf409abd731b230c217360bb3b13d5ce Sep 14 04:46:32.749781 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.749732 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb332df2d_27ca_4bb8_baad_7aadb1148973.slice/crio-82b72700ac3034f846f2301b712ce839a9230019cc35c8703c63e2c29962428f WatchSource:0}: Error finding container 82b72700ac3034f846f2301b712ce839a9230019cc35c8703c63e2c29962428f: Status 404 returned error can't find the container with id 82b72700ac3034f846f2301b712ce839a9230019cc35c8703c63e2c29962428f Sep 14 04:46:32.752407 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.752376 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc9d3cf_a117_45e4_9ee2_ebc5ebcf1340.slice/crio-6745f7f2e0b8c05dff7680fe60aee8b423e3eca9617bdfd0fcefd24f31f07677 WatchSource:0}: Error finding container 6745f7f2e0b8c05dff7680fe60aee8b423e3eca9617bdfd0fcefd24f31f07677: Status 404 returned error can't find the container with id 6745f7f2e0b8c05dff7680fe60aee8b423e3eca9617bdfd0fcefd24f31f07677 Sep 14 04:46:32.753316 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.753271 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba67880_ddb9_4e90_887e_fca7b4d93b0f.slice/crio-c4349c6bd76e615f3a07d7b18a3c508c9f61ef9e3a1409420c19d5e655075608 WatchSource:0}: Error finding container c4349c6bd76e615f3a07d7b18a3c508c9f61ef9e3a1409420c19d5e655075608: Status 404 returned error can't find the container with id c4349c6bd76e615f3a07d7b18a3c508c9f61ef9e3a1409420c19d5e655075608 Sep 14 04:46:32.755101 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:46:32.754828 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae1c54a_ac2d_4f55_ad47_7ae478fd466e.slice/crio-348e2e879f8db4036bd6163b7b95d97de9c54e42657c638da0f6647472f504d3 WatchSource:0}: Error finding container 348e2e879f8db4036bd6163b7b95d97de9c54e42657c638da0f6647472f504d3: Status 404 returned error can't find the container with id 348e2e879f8db4036bd6163b7b95d97de9c54e42657c638da0f6647472f504d3 Sep 14 04:46:33.084701 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.084506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:33.084819 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.084666 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:33.084819 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.084779 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:46:35.084763959 +0000 UTC m=+6.042380675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:33.185891 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.185848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:33.186047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.185933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:33.186047 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186007 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:33.186047 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186025 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:33.186047 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186035 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:33.186047 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186039 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:33.186286 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186088 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:35.186072852 +0000 UTC m=+6.143689567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:33.186286 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.186102 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:35.186095501 +0000 UTC m=+6.143712217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:33.498944 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.498808 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 04:41:32 +0000 UTC" deadline="2028-02-15 05:50:46.290739095 +0000 UTC" Sep 14 04:46:33.498944 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.498866 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12457h4m12.791877639s" Sep 14 04:46:33.514106 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.512362 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 04:46:33.547702 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.547412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:33.547702 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.547550 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:33.548332 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.548045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:33.548332 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:33.548222 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:33.569853 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.569818 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwbjw" event={"ID":"b10002bc-68a5-479a-a081-6e7f8a6cccb4","Type":"ContainerStarted","Data":"c14fbe86e4fed22da65230c42505885d3f619ac5be7508e3380539f5648c9a0c"} Sep 14 04:46:33.584229 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.578367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" event={"ID":"729843ab636a52a35ed9dc96e1e6fe1d","Type":"ContainerStarted","Data":"babf055e72f7a44444f5b81df3d5757bc49ddb7c80c452057396df3638f59ae5"} Sep 14 04:46:33.584229 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.582089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpvt7" event={"ID":"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340","Type":"ContainerStarted","Data":"6745f7f2e0b8c05dff7680fe60aee8b423e3eca9617bdfd0fcefd24f31f07677"} Sep 14 04:46:33.585225 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.585170 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-445p5" event={"ID":"f3371066-7823-4f23-95db-b5d47a1ce426","Type":"ContainerStarted","Data":"6a10aff500668ed075b1f55bcfe12eb522c37c2be4f0db5447d54f68be5f31b6"} Sep 14 04:46:33.589115 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.589084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerStarted","Data":"181c17f60a03866fe2f394686ceee5904e98963b78c015ab0321bc08d69a5a7e"} Sep 14 04:46:33.595094 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.595035 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-238.ec2.internal" podStartSLOduration=2.595017765 podStartE2EDuration="2.595017765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:33.594780904 +0000 UTC m=+4.552397645" watchObservedRunningTime="2026-09-14 04:46:33.595017765 +0000 UTC m=+4.552634505" Sep 14 04:46:33.603352 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.603324 2581 generic.go:356] "Generic (PLEG): container finished" podID="a417062cfd8d99b4997d23d1a6f32503" containerID="e077eb8a3f87dee63465e05968813ccccebab7a3c073ecaae6f36d841adab9fc" exitCode=0 Sep 14 04:46:33.603454 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.603411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" event={"ID":"a417062cfd8d99b4997d23d1a6f32503","Type":"ContainerDied","Data":"e077eb8a3f87dee63465e05968813ccccebab7a3c073ecaae6f36d841adab9fc"} Sep 14 04:46:33.616316 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.616212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7964" event={"ID":"b332df2d-27ca-4bb8-baad-7aadb1148973","Type":"ContainerStarted","Data":"82b72700ac3034f846f2301b712ce839a9230019cc35c8703c63e2c29962428f"} Sep 14 04:46:33.622833 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.622804 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spjtp" event={"ID":"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e","Type":"ContainerStarted","Data":"348e2e879f8db4036bd6163b7b95d97de9c54e42657c638da0f6647472f504d3"} Sep 14 04:46:33.631232 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.630810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"c4349c6bd76e615f3a07d7b18a3c508c9f61ef9e3a1409420c19d5e655075608"} Sep 14 04:46:33.643115 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.643043 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" event={"ID":"10863011-efa7-4c2b-95eb-dba333bd1ee2","Type":"ContainerStarted","Data":"01cd4c833cf6d1e1fada5cbeb9ceaa7adf409abd731b230c217360bb3b13d5ce"} Sep 14 04:46:33.659883 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:33.659822 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jtpdv" event={"ID":"a88d88d6-16f8-4645-9d28-1db5b39be0a9","Type":"ContainerStarted","Data":"09540b7ca215fe4ca1e9caf7c4a5b6277a1e805ee8a5ef80b4687308893fe936"} Sep 14 04:46:34.546412 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:34.546374 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:34.547035 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:34.546551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:34.691342 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:34.691266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" event={"ID":"a417062cfd8d99b4997d23d1a6f32503","Type":"ContainerStarted","Data":"47e43d5e7c8eed1fd0458ec811750776ba0dd99b9dbff371a2434e506ec1f853"} Sep 14 04:46:35.102754 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:35.102715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:35.102960 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.102868 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:35.102960 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.102935 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:46:39.102917721 +0000 UTC m=+10.060534454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:35.204070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:35.204029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:35.204285 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:35.204099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:35.204358 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.204318 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:35.204358 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.204344 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:35.204483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.204363 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:35.204483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.204459 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:39.204424848 +0000 UTC m=+10.162041571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:35.204963 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.204940 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:35.205055 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.205000 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:39.204983993 +0000 UTC m=+10.162600711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:35.548775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:35.548693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:35.549333 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.548826 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:35.549333 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:35.549247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:35.549468 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:35.549349 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:36.546888 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:36.546843 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:36.547090 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:36.547008 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:37.546731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:37.546698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:37.547234 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:37.546698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:37.547234 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:37.546843 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:37.547234 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:37.546928 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:38.546375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:38.546326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:38.546586 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:38.546488 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:39.147896 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:39.147843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:39.148514 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.148024 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:39.148514 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.148101 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:46:47.148079709 +0000 UTC m=+18.105696433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:39.248628 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:39.248596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:39.248816 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:39.248651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:39.248816 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248777 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:39.248816 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248793 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:39.248816 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248814 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:39.249021 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248827 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:39.249021 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248860 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:47.248838299 +0000 UTC m=+18.206455019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:39.249021 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.248881 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:47.248871966 +0000 UTC m=+18.206488691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:39.547109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:39.547006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:39.547289 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.547133 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:39.547289 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:39.547155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:39.547289 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:39.547267 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:40.547025 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:40.546976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:40.547558 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:40.547141 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:41.546403 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:41.546371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:41.546597 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:41.546497 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:41.547047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:41.546899 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:41.547047 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:41.546997 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:42.546329 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:42.546295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:42.546492 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:42.546427 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:43.546178 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:43.546146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:43.546533 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:43.546245 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:43.546533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:43.546273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:43.546533 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:43.546322 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:44.546739 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:44.546701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:44.547156 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:44.546839 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:45.546989 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:45.546958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:45.547417 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:45.547086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:45.547417 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:45.547146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:45.547417 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:45.547263 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:46.546366 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:46.546330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:46.546591 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:46.546469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:47.208475 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:47.208442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:47.208822 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.208576 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:47.208822 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.208640 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:03.208625063 +0000 UTC m=+34.166241780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:46:47.309599 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:47.309519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:47.309744 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:47.309602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:47.309744 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309702 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:47.309744 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309713 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:46:47.309744 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309732 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:46:47.309744 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309741 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:47.309935 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309753 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:03.309738792 +0000 UTC m=+34.267355517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:46:47.309935 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.309782 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:03.309765569 +0000 UTC m=+34.267382284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:46:47.546887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:47.546859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:47.547051 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.546979 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:47.547051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:47.547004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:47.547181 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:47.547139 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:48.546098 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:48.546066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:48.546549 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:48.546210 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:49.547055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:49.547022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:49.547461 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:49.547200 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:49.547461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:49.547224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:49.547461 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:49.547322 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:50.546794 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:50.546771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:50.546904 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:50.546871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:50.727907 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:50.726926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spjtp" event={"ID":"6ae1c54a-ac2d-4f55-ad47-7ae478fd466e","Type":"ContainerStarted","Data":"60e0322669b653bd8c09947b3aa8ceaf2104962cab6801a5e40cc955187d8276"} Sep 14 04:46:50.751841 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:50.751797 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-238.ec2.internal" podStartSLOduration=19.751776018 podStartE2EDuration="19.751776018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:34.707352111 +0000 UTC m=+5.664968849" watchObservedRunningTime="2026-09-14 04:46:50.751776018 +0000 UTC m=+21.709392757" Sep 14 04:46:51.547328 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.547029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:51.547570 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.547098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:51.547570 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:51.547384 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:51.547570 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:51.547447 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:51.730813 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.730776 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7964" event={"ID":"b332df2d-27ca-4bb8-baad-7aadb1148973","Type":"ContainerStarted","Data":"0591e386ad338439219d267839c31b276d360910994c821ad1cd75ace3321e17"} Sep 14 04:46:51.733887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.733862 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"c107536706a79a3be541fb0f75e42d25293ff103695344457771d77e1a283486"} Sep 14 04:46:51.734000 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.733936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"2eadba788f9897e48421b010f6bbc598f9c3be480f6f73c6434a53cfb84e2360"} Sep 14 04:46:51.734000 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.733990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"c70a27c6f184641ee59c1dbdf2c3396874acbae81f4b706fde868fb8f55494ef"} Sep 14 04:46:51.734073 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.734041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"27d9de72bae570809e577c0d28c7c15ffa58f672d675b79dbfabc7809858ebe4"} Sep 14 04:46:51.734106 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.734085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"3d4f35fa3cf8ff53f57280f46b48472128b70e9f220373a4115357d48147d3d9"} Sep 14 04:46:51.734166 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.734129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"43db6d3bc7eae5a12b158a27d7e0cc6f5697fe24241f004399759f158fda4652"} Sep 14 04:46:51.735299 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.735278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" event={"ID":"10863011-efa7-4c2b-95eb-dba333bd1ee2","Type":"ContainerStarted","Data":"919e55df1f4792ed4c62dd4665470673c0de01a0fba70a9008d63c2d995c9b8c"} Sep 14 04:46:51.736710 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.736671 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jtpdv" event={"ID":"a88d88d6-16f8-4645-9d28-1db5b39be0a9","Type":"ContainerStarted","Data":"cb8f9e32fb4d69925e37375b1b53391f6aef193d94b97f0cb186fae3ac545831"} Sep 14 04:46:51.738096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.738077 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwbjw" event={"ID":"b10002bc-68a5-479a-a081-6e7f8a6cccb4","Type":"ContainerStarted","Data":"b7179cd60b7d20555548824a758493e47d733e4a98fb9a3921e2e70302d3df5d"} Sep 14 04:46:51.739353 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.739332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-445p5" event={"ID":"f3371066-7823-4f23-95db-b5d47a1ce426","Type":"ContainerStarted","Data":"c4ac3ebca9b1861a77f4ca50d3599c31f9d6673163d36bf0dd46c67e45d3c52e"} Sep 14 04:46:51.740688 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.740670 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="db9cad919a34f5fcb4c807d216650bfb5a7096cbec354fb36d863f17541a50d9" exitCode=0 Sep 14 04:46:51.740772 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.740729 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"db9cad919a34f5fcb4c807d216650bfb5a7096cbec354fb36d863f17541a50d9"} Sep 14 04:46:51.755126 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.755072 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m7964" podStartSLOduration=3.929139062 podStartE2EDuration="21.755060842s" totalImagesPullingTime="17.82592178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:51.75504816 +0000 UTC m=+22.712664896" watchObservedRunningTime="2026-09-14 04:46:51.755060842 +0000 UTC m=+22.712677608" Sep 14 04:46:51.755256 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.755236 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-spjtp" podStartSLOduration=3.934354371 podStartE2EDuration="21.755230518s" totalImagesPullingTime="17.820876147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:50.751948364 +0000 UTC m=+21.709565113" watchObservedRunningTime="2026-09-14 04:46:51.755230518 +0000 UTC m=+22.712847315" Sep 14 04:46:51.778931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.778879 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pwbjw" podStartSLOduration=3.939451589 podStartE2EDuration="21.778864789s" totalImagesPullingTime="17.8394132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:51.778852276 +0000 UTC m=+22.736469012" watchObservedRunningTime="2026-09-14 04:46:51.778864789 +0000 UTC m=+22.736481524" Sep 14 04:46:51.825039 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.824943 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-445p5" podStartSLOduration=3.996750102 podStartE2EDuration="21.824924925s" totalImagesPullingTime="17.828174823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:51.824851445 +0000 UTC m=+22.782468182" watchObservedRunningTime="2026-09-14 04:46:51.824924925 +0000 UTC m=+22.782541662" Sep 14 04:46:51.853716 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:51.853659 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jtpdv" podStartSLOduration=3.96444457 podStartE2EDuration="21.853642841s" totalImagesPullingTime="17.889198271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:51.853629302 +0000 UTC m=+22.811246036" watchObservedRunningTime="2026-09-14 04:46:51.853642841 +0000 UTC m=+22.811259576" Sep 14 04:46:52.006178 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.006150 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 04:46:52.508361 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.508254 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T04:46:52.006172805Z","UUID":"43e75bab-5f4f-4932-94e7-c9acbe561a34","Handler":null,"Name":"","Endpoint":""} Sep 14 04:46:52.510230 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.510204 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 04:46:52.510349 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.510246 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 04:46:52.546133 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.546095 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:52.546267 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:52.546242 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:52.744300 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.744193 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpvt7" event={"ID":"bfc9d3cf-a117-45e4-9ee2-ebc5ebcf1340","Type":"ContainerStarted","Data":"0e793e34c2b88e7ee19e339db1a8658c7b82ac709c9f842bfbd4123804a67d31"} Sep 14 04:46:52.746251 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.746189 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" event={"ID":"10863011-efa7-4c2b-95eb-dba333bd1ee2","Type":"ContainerStarted","Data":"0bbd2faedac84db3a71bdff2f94513312cb1918d14dc493d5544f3abdd78c2bc"} Sep 14 04:46:52.749698 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.749676 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:52.750422 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.750407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:52.819209 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:52.819094 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wpvt7" podStartSLOduration=4.998947658 podStartE2EDuration="22.819077642s" totalImagesPullingTime="17.820129984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:52.790111789 +0000 UTC m=+23.747728528" watchObservedRunningTime="2026-09-14 04:46:52.819077642 +0000 UTC m=+23.776694432" Sep 14 04:46:53.546907 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.546823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:53.547069 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:53.546955 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:53.547069 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.547013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:53.547213 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:53.547153 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:53.751416 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.751379 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"d58587711108ed900205dd3af38a81cd1a09dc935320b7ae6cfb3bbb6fa79b76"} Sep 14 04:46:53.753433 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.753394 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" event={"ID":"10863011-efa7-4c2b-95eb-dba333bd1ee2","Type":"ContainerStarted","Data":"74e1bf8fce56e6893f7dc8688c981f2dfb7df3fe8928d3791a50c50779e407b7"} Sep 14 04:46:53.753781 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.753761 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:53.754062 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.754040 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m7964" Sep 14 04:46:53.818302 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:53.818201 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbrg8" podStartSLOduration=3.683509607 podStartE2EDuration="23.81818406s" totalImagesPullingTime="20.134674453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:53.791299297 +0000 UTC m=+24.748916038" watchObservedRunningTime="2026-09-14 04:46:53.81818406 +0000 UTC m=+24.775800797" Sep 14 04:46:54.546761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:54.546531 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:54.546933 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:54.546872 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:55.546574 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:55.546548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:55.546964 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:55.546548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:55.546964 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:55.546661 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:55.546964 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:55.546721 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:55.761745 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:55.761666 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" event={"ID":"6ba67880-ddb9-4e90-887e-fca7b4d93b0f","Type":"ContainerStarted","Data":"1abbc8429fdec5ae9dbcfbb0ec1b4b3fe5496a1ed8793ae0870ae97c8fdb25ba"} Sep 14 04:46:55.801326 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:55.801278 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" podStartSLOduration=7.92392829 podStartE2EDuration="25.80126131s" totalImagesPullingTime="17.87733302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:46:55.798300178 +0000 UTC m=+26.755916913" watchObservedRunningTime="2026-09-14 04:46:55.80126131 +0000 UTC m=+26.758878045" Sep 14 04:46:56.546774 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.546737 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:56.547161 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:56.546858 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:56.764934 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.764893 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:56.765090 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.764997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:56.765090 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.765045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:56.781215 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.781185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:56.781624 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:56.781591 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:46:57.339386 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.339354 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lw9b4"] Sep 14 04:46:57.339536 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.339519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:57.339666 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:57.339641 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:57.341645 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.341607 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbv9t"] Sep 14 04:46:57.341723 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.341702 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:57.341800 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:57.341780 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:57.345351 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.345327 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xdm24"] Sep 14 04:46:57.345454 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.345441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:57.345567 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:57.345549 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:57.768569 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.768535 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="2cad9ca1f97e27f5525349b21cd89f29001d73ef6705da7858c9b2600a0622cd" exitCode=0 Sep 14 04:46:57.768991 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:57.768618 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"2cad9ca1f97e27f5525349b21cd89f29001d73ef6705da7858c9b2600a0622cd"} Sep 14 04:46:58.772219 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:58.772156 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="5c53a07bae573cce3477c80d8fe7a538f07241822be47310891b54be46598ad4" exitCode=0 Sep 14 04:46:58.772701 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:58.772234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"5c53a07bae573cce3477c80d8fe7a538f07241822be47310891b54be46598ad4"} Sep 14 04:46:59.547416 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:59.547392 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:46:59.547526 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:59.547453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:46:59.547591 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:59.547545 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:46:59.547647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:59.547631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:46:59.547742 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:59.547717 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:46:59.547807 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:46:59.547791 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:46:59.777343 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:59.777267 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="a34f71ee9bffce307b57eafb3aa919b138fded6fb53f941a7a9ad4ca69f4294e" exitCode=0 Sep 14 04:46:59.777343 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:46:59.777305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"a34f71ee9bffce307b57eafb3aa919b138fded6fb53f941a7a9ad4ca69f4294e"} Sep 14 04:47:01.546856 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:01.546828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:01.546856 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:01.546843 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:01.547272 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:01.546828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:47:01.547272 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:01.546943 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbv9t" podUID="64283daa-b49e-402f-b79a-82fad7bdf8a8" Sep 14 04:47:01.547272 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:01.547036 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:47:01.547272 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:01.547137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lw9b4" podUID="2e465651-931b-4cc0-9fc3-f3eee8ce3ebd" Sep 14 04:47:03.225018 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.224982 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:03.225452 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.225111 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 04:47:03.225452 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.225183 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret podName:2e465651-931b-4cc0-9fc3-f3eee8ce3ebd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:35.225163559 +0000 UTC m=+66.182780288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret") pod "global-pull-secret-syncer-lw9b4" (UID: "2e465651-931b-4cc0-9fc3-f3eee8ce3ebd") : object "kube-system"/"original-pull-secret" not registered Sep 14 04:47:03.306842 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.306814 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-238.ec2.internal" event="NodeReady" Sep 14 04:47:03.307011 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.306992 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 04:47:03.325745 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.325711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:47:03.325871 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.325773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:03.325871 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325849 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:47:03.325970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325872 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 04:47:03.325970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325888 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 04:47:03.325970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325897 2581 projected.go:196] Error preparing data for projected volume kube-api-access-n4t4p for pod openshift-network-diagnostics/network-check-target-cbv9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:47:03.325970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325918 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:35.325899247 +0000 UTC m=+66.283515975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 04:47:03.325970 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.325939 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p podName:64283daa-b49e-402f-b79a-82fad7bdf8a8 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:35.325929679 +0000 UTC m=+66.283546394 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4t4p" (UniqueName: "kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p") pod "network-check-target-cbv9t" (UID: "64283daa-b49e-402f-b79a-82fad7bdf8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 04:47:03.358345 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.358314 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:47:03.449030 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.448960 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:47:03.449030 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.449001 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7jcz2"] Sep 14 04:47:03.449235 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.449075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.452658 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.452631 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 04:47:03.453043 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.452940 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 04:47:03.453155 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.453076 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgrqm\"" Sep 14 04:47:03.459788 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.459765 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 04:47:03.460570 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.460554 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 04:47:03.463229 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.463207 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-479w5"] Sep 14 04:47:03.463328 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.463312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.471765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.471728 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 04:47:03.471859 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.471785 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46brg\"" Sep 14 04:47:03.471921 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.471893 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 04:47:03.472141 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.472028 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 04:47:03.481429 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.481370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7jcz2"] Sep 14 04:47:03.481516 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.481462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.481516 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.481467 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-479w5"] Sep 14 04:47:03.486872 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.486852 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 04:47:03.487336 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.487317 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 04:47:03.487416 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.487365 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwfwc\"" Sep 14 04:47:03.546647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.546620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:03.546647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.546640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:03.546894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.546625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:47:03.549497 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.549478 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 04:47:03.549647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.549550 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 04:47:03.549763 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.549737 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgkvn\"" Sep 14 04:47:03.549763 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.549758 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 04:47:03.550499 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.550478 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 04:47:03.550994 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.550962 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54hj4\"" Sep 14 04:47:03.583060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.583032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:03.629001 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.628968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdc47c-6d0b-43e0-937e-a413fdb46d11-config-volume\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.629158 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629158 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629390 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629390 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77b7\" (UniqueName: \"kubernetes.io/projected/edd700df-95e3-489b-93a3-1e6ee162a412-kube-api-access-s77b7\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.629390 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4zn\" (UniqueName: \"kubernetes.io/projected/37cdc47c-6d0b-43e0-937e-a413fdb46d11-kube-api-access-rq4zn\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.629390 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.629566 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629412 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629566 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb65n\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.629566 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629457 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd700df-95e3-489b-93a3-1e6ee162a412-cert\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.629566 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.629480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cdc47c-6d0b-43e0-937e-a413fdb46d11-tmp-dir\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.691350 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.691314 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:03.730320 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.730320 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77b7\" (UniqueName: \"kubernetes.io/projected/edd700df-95e3-489b-93a3-1e6ee162a412-kube-api-access-s77b7\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.730320 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4zn\" (UniqueName: \"kubernetes.io/projected/37cdc47c-6d0b-43e0-937e-a413fdb46d11-kube-api-access-rq4zn\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb65n\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd700df-95e3-489b-93a3-1e6ee162a412-cert\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cdc47c-6d0b-43e0-937e-a413fdb46d11-tmp-dir\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdc47c-6d0b-43e0-937e-a413fdb46d11-config-volume\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.730591 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.731010 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cdc47c-6d0b-43e0-937e-a413fdb46d11-tmp-dir\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.731010 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.730968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.731106 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.731006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.731106 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.730403 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:03.731106 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.731066 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:03.731106 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.730460 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:03.731319 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.731190 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:04.231169922 +0000 UTC m=+35.188786641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:03.731377 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.731364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdc47c-6d0b-43e0-937e-a413fdb46d11-config-volume\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.731431 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.731388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.731482 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:03.731438 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:04.231423045 +0000 UTC m=+35.189039766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:03.732081 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.732053 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.732261 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.732239 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.735181 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.735160 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.735324 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.735305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.735375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.735363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd700df-95e3-489b-93a3-1e6ee162a412-cert\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.740623 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.740599 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.740740 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.740620 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb65n\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:03.740945 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.740923 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77b7\" (UniqueName: \"kubernetes.io/projected/edd700df-95e3-489b-93a3-1e6ee162a412-kube-api-access-s77b7\") pod \"ingress-canary-7jcz2\" (UID: \"edd700df-95e3-489b-93a3-1e6ee162a412\") " pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.744745 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.744704 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4zn\" (UniqueName: \"kubernetes.io/projected/37cdc47c-6d0b-43e0-937e-a413fdb46d11-kube-api-access-rq4zn\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:03.772863 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.772831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7jcz2" Sep 14 04:47:03.922422 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:03.922386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:04.023883 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.023811 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.235019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.235092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:04.235251 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:04.235275 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:04.235289 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:04.235346 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:05.235326763 +0000 UTC m=+36.192943478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:04.235483 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:04.235378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:05.23536818 +0000 UTC m=+36.192984895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:04.467736 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.467710 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7jcz2"] Sep 14 04:47:04.472529 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:47:04.472502 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd700df_95e3_489b_93a3_1e6ee162a412.slice/crio-c55cf06111760821113cebbc6810f65d35bc486eb6b6abea340b7be2a4c2fe57 WatchSource:0}: Error finding container c55cf06111760821113cebbc6810f65d35bc486eb6b6abea340b7be2a4c2fe57: Status 404 returned error can't find the container with id c55cf06111760821113cebbc6810f65d35bc486eb6b6abea340b7be2a4c2fe57 Sep 14 04:47:04.754504 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.754430 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:04.791435 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.791405 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="49b31b2877e4c9aafe3a5c5c0cb4d5a050a134f898fa53ddb7a67bbda94a230c" exitCode=0 Sep 14 04:47:04.791572 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.791479 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"49b31b2877e4c9aafe3a5c5c0cb4d5a050a134f898fa53ddb7a67bbda94a230c"} Sep 14 04:47:04.792708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.792688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7jcz2" event={"ID":"edd700df-95e3-489b-93a3-1e6ee162a412","Type":"ContainerStarted","Data":"c55cf06111760821113cebbc6810f65d35bc486eb6b6abea340b7be2a4c2fe57"} Sep 14 04:47:04.842507 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:04.842478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:05.243249 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:05.243211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:05.243320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:05.243370 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:05.243443 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:05.243450 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:07.243428211 +0000 UTC m=+38.201044939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:05.243457 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:05.243709 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:05.243504 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:07.243488724 +0000 UTC m=+38.201105456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:05.798983 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:05.798953 2581 generic.go:356] "Generic (PLEG): container finished" podID="66f3737b-d741-4c5c-8788-f184f954777e" containerID="209115609c7b68eb96e9d705a866e53bace2e9ad469f825af8526026a037fc34" exitCode=0 Sep 14 04:47:05.799190 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:05.799022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerDied","Data":"209115609c7b68eb96e9d705a866e53bace2e9ad469f825af8526026a037fc34"} Sep 14 04:47:06.707257 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:06.707213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:06.806438 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:06.806393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8764" event={"ID":"66f3737b-d741-4c5c-8788-f184f954777e","Type":"ContainerStarted","Data":"721ec1bec63b2bb1d2593f18c6aa1e6814ab5e8c4c79dd23f7c24ac028b946cb"} Sep 14 04:47:06.831263 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:06.831217 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j8764" podStartSLOduration=7.833123142 podStartE2EDuration="36.831202627s" totalImagesPullingTime="28.998079485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:06.829871536 +0000 UTC m=+37.787488271" watchObservedRunningTime="2026-09-14 04:47:06.831202627 +0000 UTC m=+37.788819362" Sep 14 04:47:06.955926 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:06.955902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:07.056292 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:07.056235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" limit=1024 Sep 14 04:47:07.261332 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:07.261299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:07.261490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:07.261352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:07.261490 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:07.261441 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:07.261490 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:07.261459 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:07.261490 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:07.261479 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:07.261490 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:07.261490 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:11.261477368 +0000 UTC m=+42.219094083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:07.261671 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:07.261529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:11.261511598 +0000 UTC m=+42.219128326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:07.809666 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:07.809595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7jcz2" event={"ID":"edd700df-95e3-489b-93a3-1e6ee162a412","Type":"ContainerStarted","Data":"7690da3528625f86a4377b78bf3ec54c34110f323c8d788473082226a382f502"} Sep 14 04:47:07.829302 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:07.829261 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7jcz2" podStartSLOduration=2.213455909 podStartE2EDuration="4.829249455s" totalImagesPullingTime="2.615793546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:07.828085728 +0000 UTC m=+38.785702463" watchObservedRunningTime="2026-09-14 04:47:07.829249455 +0000 UTC m=+38.786866190" Sep 14 04:47:11.289112 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:11.289066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:11.289170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:11.289218 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:11.289240 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:11.289294 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:19.28928032 +0000 UTC m=+50.246897035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:11.289301 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:11.289554 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:11.289348 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:19.289334159 +0000 UTC m=+50.246950873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:19.351752 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:19.351703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:19.351783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:19.351888 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:19.351911 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:19.351928 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:19.351976 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:35.351958721 +0000 UTC m=+66.309575444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:19.352169 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:19.351989 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:35.351984203 +0000 UTC m=+66.309600917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:28.785634 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:28.785599 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qsvh" Sep 14 04:47:35.267385 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.267338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:35.270695 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.270673 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 04:47:35.281529 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.281498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e465651-931b-4cc0-9fc3-f3eee8ce3ebd-original-pull-secret\") pod \"global-pull-secret-syncer-lw9b4\" (UID: \"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd\") " pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:35.366955 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.366923 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lw9b4" Sep 14 04:47:35.369194 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.369175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:47:35.369251 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.369211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:47:35.369251 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.369239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:47:35.369319 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.369254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:35.369371 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.369345 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:47:35.369446 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.369433 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:07.369412663 +0000 UTC m=+98.327029390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:47:35.369446 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.369357 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:47:35.369546 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.369456 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:47:35.369546 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.369491 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:07.369481809 +0000 UTC m=+98.327098538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:47:35.371540 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.371519 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 04:47:35.371651 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.371559 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 04:47:35.380361 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.380341 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 04:47:35.380430 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:47:35.380416 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:39.380398067 +0000 UTC m=+130.338014784 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : secret "metrics-daemon-secret" not found Sep 14 04:47:35.382130 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.382104 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 04:47:35.393110 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.393087 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t4p\" (UniqueName: \"kubernetes.io/projected/64283daa-b49e-402f-b79a-82fad7bdf8a8-kube-api-access-n4t4p\") pod \"network-check-target-cbv9t\" (UID: \"64283daa-b49e-402f-b79a-82fad7bdf8a8\") " pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:35.660001 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.659969 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lw9b4"] Sep 14 04:47:35.662111 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.662084 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54hj4\"" Sep 14 04:47:35.670593 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.670578 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:35.679067 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:47:35.679027 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e465651_931b_4cc0_9fc3_f3eee8ce3ebd.slice/crio-c48e8d1ca381868255c1b05400139a53258eeb96a9741eec6d969c086503699e WatchSource:0}: Error finding container c48e8d1ca381868255c1b05400139a53258eeb96a9741eec6d969c086503699e: Status 404 returned error can't find the container with id c48e8d1ca381868255c1b05400139a53258eeb96a9741eec6d969c086503699e Sep 14 04:47:35.878111 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.878071 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lw9b4" event={"ID":"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd","Type":"ContainerStarted","Data":"c48e8d1ca381868255c1b05400139a53258eeb96a9741eec6d969c086503699e"} Sep 14 04:47:35.963750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:35.963676 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbv9t"] Sep 14 04:47:35.966987 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:47:35.966965 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64283daa_b49e_402f_b79a_82fad7bdf8a8.slice/crio-fd7a2a7676472c3480a74fd33ad6dab14bc72f01d978f6fbb8aa402dcea96edb WatchSource:0}: Error finding container fd7a2a7676472c3480a74fd33ad6dab14bc72f01d978f6fbb8aa402dcea96edb: Status 404 returned error can't find the container with id fd7a2a7676472c3480a74fd33ad6dab14bc72f01d978f6fbb8aa402dcea96edb Sep 14 04:47:36.882072 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:36.881999 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbv9t" event={"ID":"64283daa-b49e-402f-b79a-82fad7bdf8a8","Type":"ContainerStarted","Data":"fd7a2a7676472c3480a74fd33ad6dab14bc72f01d978f6fbb8aa402dcea96edb"} Sep 14 04:47:40.894795 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:40.894711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lw9b4" event={"ID":"2e465651-931b-4cc0-9fc3-f3eee8ce3ebd","Type":"ContainerStarted","Data":"7d0d9d79464a208b5eb32fb631ef2a62f6936d89b6c8f4c235392524fa54b30f"} Sep 14 04:47:40.896220 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:40.896187 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbv9t" event={"ID":"64283daa-b49e-402f-b79a-82fad7bdf8a8","Type":"ContainerStarted","Data":"aa13c36449adc7c3ea2bdd086aacc602451fdc29723f8d4afb954998753bd2c6"} Sep 14 04:47:40.896328 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:40.896318 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:47:40.927025 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:47:40.926972 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lw9b4" podStartSLOduration=65.985566369 podStartE2EDuration="1m10.926955137s" totalImagesPullingTime="4.941388768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:40.911546234 +0000 UTC m=+71.869162966" watchObservedRunningTime="2026-09-14 04:47:40.926955137 +0000 UTC m=+71.884571873" Sep 14 04:48:07.389976 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:07.389936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:07.390005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:07.390085 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:07.390100 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:07.390113 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db78995bf-vw9bj: secret "image-registry-tls" not found Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:07.390185 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls podName:908c94ac-b68f-4179-9290-123bdc9a1cc1 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:11.390171798 +0000 UTC m=+162.347788513 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls") pod "image-registry-6db78995bf-vw9bj" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1") : secret "image-registry-tls" not found Sep 14 04:48:07.390427 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:07.390197 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls podName:37cdc47c-6d0b-43e0-937e-a413fdb46d11 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:11.390192076 +0000 UTC m=+162.347808791 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls") pod "dns-default-479w5" (UID: "37cdc47c-6d0b-43e0-937e-a413fdb46d11") : secret "dns-default-metrics-tls" not found Sep 14 04:48:11.901541 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:11.901509 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cbv9t" Sep 14 04:48:11.919106 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:11.919053 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cbv9t" podStartSLOduration=97.264759714 podStartE2EDuration="1m41.919036788s" totalImagesPullingTime="4.654277074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:40.926604473 +0000 UTC m=+71.884221209" watchObservedRunningTime="2026-09-14 04:48:11.919036788 +0000 UTC m=+102.876653562" Sep 14 04:48:39.419215 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:39.419150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:48:39.419712 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:39.419310 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 04:48:39.419712 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:39.419378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs podName:c054e08e-84af-4062-bd3c-4456cea1f2e5 nodeName:}" failed. No retries permitted until 2026-09-14 04:50:41.419363142 +0000 UTC m=+252.376979857 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs") pod "network-metrics-daemon-xdm24" (UID: "c054e08e-84af-4062-bd3c-4456cea1f2e5") : secret "metrics-daemon-secret" not found Sep 14 04:48:41.021566 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.021518 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b"] Sep 14 04:48:41.028087 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.028047 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww"] Sep 14 04:48:41.028268 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.028179 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.030713 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.030685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" Sep 14 04:48:41.031680 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.031663 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 14 04:48:41.031765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.031729 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 14 04:48:41.031840 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.031817 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vp6lc\"" Sep 14 04:48:41.032703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.032530 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 14 04:48:41.033368 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.033348 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 14 04:48:41.033899 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.033878 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-pkk55"] Sep 14 04:48:41.034333 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.034314 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ptvkk\"" Sep 14 04:48:41.034618 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.034401 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 14 04:48:41.035990 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.035969 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 14 04:48:41.036769 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.036753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.039241 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.039198 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww"] Sep 14 04:48:41.039755 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.039736 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jgcm5\"" Sep 14 04:48:41.039846 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.039755 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 04:48:41.040023 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.040005 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 14 04:48:41.040104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.040049 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b"] Sep 14 04:48:41.040461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.040441 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 14 04:48:41.040725 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.040710 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 04:48:41.044905 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.044881 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 14 04:48:41.052359 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.052331 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-pkk55"] Sep 14 04:48:41.131092 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bj4\" (UniqueName: \"kubernetes.io/projected/98604993-7109-4566-bbf4-9e909393f78e-kube-api-access-r7bj4\") pod \"volume-data-source-validator-86695b7cc7-qfjww\" (UID: \"98604993-7109-4566-bbf4-9e909393f78e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" Sep 14 04:48:41.131092 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-snapshots\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131139 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-trusted-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pk5m\" (UniqueName: \"kubernetes.io/projected/aa28282c-eebb-4e68-a437-97c35d9c3293-kube-api-access-5pk5m\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.131375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa28282c-eebb-4e68-a437-97c35d9c3293-serving-cert\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.131375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9g2\" (UniqueName: \"kubernetes.io/projected/e3b45323-6c9b-4747-8665-d81e85a37840-kube-api-access-sc9g2\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131565 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-tmp\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131565 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-service-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131565 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b45323-6c9b-4747-8665-d81e85a37840-serving-cert\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.131565 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.131488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa28282c-eebb-4e68-a437-97c35d9c3293-config\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.232158 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-service-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232158 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b45323-6c9b-4747-8665-d81e85a37840-serving-cert\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232158 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa28282c-eebb-4e68-a437-97c35d9c3293-config\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.232469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bj4\" (UniqueName: \"kubernetes.io/projected/98604993-7109-4566-bbf4-9e909393f78e-kube-api-access-r7bj4\") pod \"volume-data-source-validator-86695b7cc7-qfjww\" (UID: \"98604993-7109-4566-bbf4-9e909393f78e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" Sep 14 04:48:41.232469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-snapshots\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232232 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-trusted-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pk5m\" (UniqueName: \"kubernetes.io/projected/aa28282c-eebb-4e68-a437-97c35d9c3293-kube-api-access-5pk5m\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.232469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa28282c-eebb-4e68-a437-97c35d9c3293-serving-cert\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.232723 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9g2\" (UniqueName: \"kubernetes.io/projected/e3b45323-6c9b-4747-8665-d81e85a37840-kube-api-access-sc9g2\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232723 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-tmp\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-service-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.232939 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-tmp\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.233018 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa28282c-eebb-4e68-a437-97c35d9c3293-config\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.233018 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.232997 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3b45323-6c9b-4747-8665-d81e85a37840-snapshots\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.233320 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.233303 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b45323-6c9b-4747-8665-d81e85a37840-trusted-ca-bundle\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.234701 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.234675 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b45323-6c9b-4747-8665-d81e85a37840-serving-cert\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.234845 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.234829 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa28282c-eebb-4e68-a437-97c35d9c3293-serving-cert\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.241599 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.241576 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pk5m\" (UniqueName: \"kubernetes.io/projected/aa28282c-eebb-4e68-a437-97c35d9c3293-kube-api-access-5pk5m\") pod \"service-ca-operator-cf7c776b7-5c25b\" (UID: \"aa28282c-eebb-4e68-a437-97c35d9c3293\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.241856 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.241839 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9g2\" (UniqueName: \"kubernetes.io/projected/e3b45323-6c9b-4747-8665-d81e85a37840-kube-api-access-sc9g2\") pod \"insights-operator-5844b856db-pkk55\" (UID: \"e3b45323-6c9b-4747-8665-d81e85a37840\") " pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.242011 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.241989 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bj4\" (UniqueName: \"kubernetes.io/projected/98604993-7109-4566-bbf4-9e909393f78e-kube-api-access-r7bj4\") pod \"volume-data-source-validator-86695b7cc7-qfjww\" (UID: \"98604993-7109-4566-bbf4-9e909393f78e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" Sep 14 04:48:41.339878 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.339783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" Sep 14 04:48:41.346889 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.346870 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" Sep 14 04:48:41.351695 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.351653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-pkk55" Sep 14 04:48:41.625867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.625836 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww"] Sep 14 04:48:41.629136 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.629094 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b"] Sep 14 04:48:41.629845 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:48:41.629806 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98604993_7109_4566_bbf4_9e909393f78e.slice/crio-886dc59e27bcd6a34da69258f8acfe5b03b88ae31315c2ba4701376ad66d2c25 WatchSource:0}: Error finding container 886dc59e27bcd6a34da69258f8acfe5b03b88ae31315c2ba4701376ad66d2c25: Status 404 returned error can't find the container with id 886dc59e27bcd6a34da69258f8acfe5b03b88ae31315c2ba4701376ad66d2c25 Sep 14 04:48:41.630009 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:41.629957 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-pkk55"] Sep 14 04:48:41.633232 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:48:41.633206 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b45323_6c9b_4747_8665_d81e85a37840.slice/crio-f359a4cc9eda9bc1fc919570ab3342e85e3c7c0678a1298a05f6d7e6e97a4b41 WatchSource:0}: Error finding container f359a4cc9eda9bc1fc919570ab3342e85e3c7c0678a1298a05f6d7e6e97a4b41: Status 404 returned error can't find the container with id f359a4cc9eda9bc1fc919570ab3342e85e3c7c0678a1298a05f6d7e6e97a4b41 Sep 14 04:48:41.633972 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:48:41.633952 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa28282c_eebb_4e68_a437_97c35d9c3293.slice/crio-b150d9855fc08b15cba244909e25029e72ed080dbbfa0fd75e24605febe8fb67 WatchSource:0}: Error finding container b150d9855fc08b15cba244909e25029e72ed080dbbfa0fd75e24605febe8fb67: Status 404 returned error can't find the container with id b150d9855fc08b15cba244909e25029e72ed080dbbfa0fd75e24605febe8fb67 Sep 14 04:48:42.039269 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:42.039165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" event={"ID":"aa28282c-eebb-4e68-a437-97c35d9c3293","Type":"ContainerStarted","Data":"b150d9855fc08b15cba244909e25029e72ed080dbbfa0fd75e24605febe8fb67"} Sep 14 04:48:42.042728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:42.042679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" event={"ID":"98604993-7109-4566-bbf4-9e909393f78e","Type":"ContainerStarted","Data":"886dc59e27bcd6a34da69258f8acfe5b03b88ae31315c2ba4701376ad66d2c25"} Sep 14 04:48:42.044193 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:42.044167 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-pkk55" event={"ID":"e3b45323-6c9b-4747-8665-d81e85a37840","Type":"ContainerStarted","Data":"f359a4cc9eda9bc1fc919570ab3342e85e3c7c0678a1298a05f6d7e6e97a4b41"} Sep 14 04:48:43.049239 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:43.049201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" event={"ID":"98604993-7109-4566-bbf4-9e909393f78e","Type":"ContainerStarted","Data":"c25fa4853afc6fb6d6890eea08c0d4ef27150c49356502f6eb74d2516cd53452"} Sep 14 04:48:43.068382 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:43.068313 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qfjww" podStartSLOduration=1.772213309 podStartE2EDuration="3.068295925s" totalImagesPullingTime="1.296082616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:43.067944999 +0000 UTC m=+134.025561737" watchObservedRunningTime="2026-09-14 04:48:43.068295925 +0000 UTC m=+134.025912662" Sep 14 04:48:45.030656 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.030618 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hdq5h"] Sep 14 04:48:45.033789 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.033771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.035708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.035682 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 04:48:45.035856 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.035684 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8h7x\"" Sep 14 04:48:45.035999 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.035979 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 04:48:45.053486 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.053463 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdq5h"] Sep 14 04:48:45.061658 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.061627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-pkk55" event={"ID":"e3b45323-6c9b-4747-8665-d81e85a37840","Type":"ContainerStarted","Data":"446820d03d612232cebbd0c5a75f58cc223fb331438b09267d49fb4cd740f7d4"} Sep 14 04:48:45.063093 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.063070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" event={"ID":"aa28282c-eebb-4e68-a437-97c35d9c3293","Type":"ContainerStarted","Data":"bba80976edc88f87ca3d080178668460b1a800ab0501092ef7f5c78109a0515c"} Sep 14 04:48:45.113585 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.113522 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" podStartSLOduration=2.756737336 podStartE2EDuration="5.113499967s" totalImagesPullingTime="2.356762631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:45.112797603 +0000 UTC m=+136.070414340" watchObservedRunningTime="2026-09-14 04:48:45.113499967 +0000 UTC m=+136.071116707" Sep 14 04:48:45.133663 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.133600 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-pkk55" podStartSLOduration=2.774484376 podStartE2EDuration="5.13357767s" totalImagesPullingTime="2.359093294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:45.131858762 +0000 UTC m=+136.089475499" watchObservedRunningTime="2026-09-14 04:48:45.13357767 +0000 UTC m=+136.091194408" Sep 14 04:48:45.163600 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.163566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0273b0e-9540-4ef2-adee-fc134683dcd3-data-volume\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.163765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.163714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0273b0e-9540-4ef2-adee-fc134683dcd3-crio-socket\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.163765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.163753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.163878 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.163855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vpb\" (UniqueName: \"kubernetes.io/projected/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-api-access-77vpb\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.164003 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.163987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vpb\" (UniqueName: \"kubernetes.io/projected/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-api-access-77vpb\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0273b0e-9540-4ef2-adee-fc134683dcd3-data-volume\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0273b0e-9540-4ef2-adee-fc134683dcd3-crio-socket\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.265186 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.265265 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:45.765245644 +0000 UTC m=+136.722862385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : configmap "kube-rbac-proxy" not found Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0273b0e-9540-4ef2-adee-fc134683dcd3-crio-socket\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.265340 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.265320 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:48:45.265739 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.265363 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:45.765349827 +0000 UTC m=+136.722966556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:48:45.265739 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.265510 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0273b0e-9540-4ef2-adee-fc134683dcd3-data-volume\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.275412 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.275384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vpb\" (UniqueName: \"kubernetes.io/projected/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-api-access-77vpb\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.769470 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.769422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.769678 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:45.769500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:45.769678 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.769620 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 04:48:45.769678 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.769661 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:48:45.769850 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.769709 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:46.769688009 +0000 UTC m=+137.727304728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : configmap "kube-rbac-proxy" not found Sep 14 04:48:45.769850 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:45.769729 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:46.769719336 +0000 UTC m=+137.727336052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:48:46.352383 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.352348 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2"] Sep 14 04:48:46.355499 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.355483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" Sep 14 04:48:46.357642 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.357623 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hj2jk\"" Sep 14 04:48:46.362554 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.362530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2"] Sep 14 04:48:46.476559 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.476505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfx6l\" (UniqueName: \"kubernetes.io/projected/921c03cd-b0e8-47e0-9891-902cb6bec81e-kube-api-access-hfx6l\") pod \"network-check-source-6f8b768455-ptzr2\" (UID: \"921c03cd-b0e8-47e0-9891-902cb6bec81e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" Sep 14 04:48:46.577221 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.577182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfx6l\" (UniqueName: \"kubernetes.io/projected/921c03cd-b0e8-47e0-9891-902cb6bec81e-kube-api-access-hfx6l\") pod \"network-check-source-6f8b768455-ptzr2\" (UID: \"921c03cd-b0e8-47e0-9891-902cb6bec81e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" Sep 14 04:48:46.585112 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.585084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfx6l\" (UniqueName: \"kubernetes.io/projected/921c03cd-b0e8-47e0-9891-902cb6bec81e-kube-api-access-hfx6l\") pod \"network-check-source-6f8b768455-ptzr2\" (UID: \"921c03cd-b0e8-47e0-9891-902cb6bec81e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" Sep 14 04:48:46.665135 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.665025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" Sep 14 04:48:46.778540 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.778503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:46.778700 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.778555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:46.778700 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:46.778627 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 04:48:46.778700 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:46.778680 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.778667811 +0000 UTC m=+139.736284526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : configmap "kube-rbac-proxy" not found Sep 14 04:48:46.778700 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:46.778676 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:48:46.778921 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:46.778752 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.77873175 +0000 UTC m=+139.736348469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:48:46.940946 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.940857 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2"] Sep 14 04:48:46.945554 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:48:46.945509 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921c03cd_b0e8_47e0_9891_902cb6bec81e.slice/crio-b4b6ce37ac0b1679a46a76b34c0c4e506e57675a968aee3387a7944ec888a4bc WatchSource:0}: Error finding container b4b6ce37ac0b1679a46a76b34c0c4e506e57675a968aee3387a7944ec888a4bc: Status 404 returned error can't find the container with id b4b6ce37ac0b1679a46a76b34c0c4e506e57675a968aee3387a7944ec888a4bc Sep 14 04:48:46.974954 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:46.974935 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwbjw_b10002bc-68a5-479a-a081-6e7f8a6cccb4/dns-node-resolver/0.log" Sep 14 04:48:47.070686 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:47.070651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" event={"ID":"921c03cd-b0e8-47e0-9891-902cb6bec81e","Type":"ContainerStarted","Data":"20bf3c25d69e86bdbfa5463271cf6d55c82ff4e9a79030fe02a76ec0cc1bfbab"} Sep 14 04:48:47.070876 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:47.070701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" event={"ID":"921c03cd-b0e8-47e0-9891-902cb6bec81e","Type":"ContainerStarted","Data":"b4b6ce37ac0b1679a46a76b34c0c4e506e57675a968aee3387a7944ec888a4bc"} Sep 14 04:48:47.088152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:47.088079 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptzr2" podStartSLOduration=1.08806474 podStartE2EDuration="1.08806474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:47.086741177 +0000 UTC m=+138.044357916" watchObservedRunningTime="2026-09-14 04:48:47.08806474 +0000 UTC m=+138.045681477" Sep 14 04:48:47.805457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:47.805428 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-445p5_f3371066-7823-4f23-95db-b5d47a1ce426/node-ca/0.log" Sep 14 04:48:48.578807 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:48.578769 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7jcz2_edd700df-95e3-489b-93a3-1e6ee162a412/serve-healthcheck-canary/0.log" Sep 14 04:48:48.791917 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:48.791876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:48.792093 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:48.791939 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:48.792093 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:48.792010 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 04:48:48.792093 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:48.792023 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:48:48.792093 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:48.792079 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:52.792063938 +0000 UTC m=+143.749680653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:48:48.792093 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:48.792093 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:52.792086338 +0000 UTC m=+143.749703052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : configmap "kube-rbac-proxy" not found Sep 14 04:48:52.826337 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:52.826289 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:52.826798 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:48:52.826358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:48:52.826798 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:52.826455 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 04:48:52.826798 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:52.826458 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:48:52.826798 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:52.826534 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:00.826513004 +0000 UTC m=+151.784129741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : configmap "kube-rbac-proxy" not found Sep 14 04:48:52.826798 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:48:52.826555 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:00.826545366 +0000 UTC m=+151.784162081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:49:00.884963 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:00.884899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:00.885415 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:00.884995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:00.885415 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:00.885055 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 04:49:00.885415 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:00.885143 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls podName:c0273b0e-9540-4ef2-adee-fc134683dcd3 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:16.885106068 +0000 UTC m=+167.842722787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdq5h" (UID: "c0273b0e-9540-4ef2-adee-fc134683dcd3") : secret "insights-runtime-extractor-tls" not found Sep 14 04:49:00.885577 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:00.885556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0273b0e-9540-4ef2-adee-fc134683dcd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:06.460224 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:06.460178 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" Sep 14 04:49:06.491380 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:06.491344 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-479w5" podUID="37cdc47c-6d0b-43e0-937e-a413fdb46d11" Sep 14 04:49:06.572555 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:06.572513 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xdm24" podUID="c054e08e-84af-4062-bd3c-4456cea1f2e5" Sep 14 04:49:07.126819 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:07.126788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-479w5" Sep 14 04:49:11.467746 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.467701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:11.468182 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.467757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:49:11.470078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.470058 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37cdc47c-6d0b-43e0-937e-a413fdb46d11-metrics-tls\") pod \"dns-default-479w5\" (UID: \"37cdc47c-6d0b-43e0-937e-a413fdb46d11\") " pod="openshift-dns/dns-default-479w5" Sep 14 04:49:11.470226 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.470205 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"image-registry-6db78995bf-vw9bj\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:11.629715 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.629684 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwfwc\"" Sep 14 04:49:11.638316 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.638296 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-479w5" Sep 14 04:49:11.906409 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:11.906378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-479w5"] Sep 14 04:49:11.909527 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:11.909490 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cdc47c_6d0b_43e0_937e_a413fdb46d11.slice/crio-321d94da05e8b68a69f9e1b3e0b53ea407476d5f3576fc2d7814b96fcb75c5b7 WatchSource:0}: Error finding container 321d94da05e8b68a69f9e1b3e0b53ea407476d5f3576fc2d7814b96fcb75c5b7: Status 404 returned error can't find the container with id 321d94da05e8b68a69f9e1b3e0b53ea407476d5f3576fc2d7814b96fcb75c5b7 Sep 14 04:49:12.142767 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:12.142730 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-479w5" event={"ID":"37cdc47c-6d0b-43e0-937e-a413fdb46d11","Type":"ContainerStarted","Data":"321d94da05e8b68a69f9e1b3e0b53ea407476d5f3576fc2d7814b96fcb75c5b7"} Sep 14 04:49:14.151474 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:14.151443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-479w5" event={"ID":"37cdc47c-6d0b-43e0-937e-a413fdb46d11","Type":"ContainerStarted","Data":"c98e693092aeac07f48b0fb75cd5b298fc1a3188e477db950cd3f1ec253843b4"} Sep 14 04:49:14.151838 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:14.151493 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-479w5" event={"ID":"37cdc47c-6d0b-43e0-937e-a413fdb46d11","Type":"ContainerStarted","Data":"e0663cdb204187641c6fa174059b3ab7987979ff2438885d8e98200886d11a3d"} Sep 14 04:49:14.151838 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:14.151578 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-479w5" Sep 14 04:49:14.168523 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:14.168465 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-479w5" podStartSLOduration=129.412349797 podStartE2EDuration="2m11.16844161s" totalImagesPullingTime="1.756091813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:14.167895207 +0000 UTC m=+165.125511944" watchObservedRunningTime="2026-09-14 04:49:14.16844161 +0000 UTC m=+165.126058348" Sep 14 04:49:15.360907 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.360875 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s"] Sep 14 04:49:15.363887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.363870 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:15.365662 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.365637 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vbxrd\"" Sep 14 04:49:15.366069 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.366048 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 14 04:49:15.378365 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.378342 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s"] Sep 14 04:49:15.499492 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.499449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db24ee8c-1786-4098-b19a-f6c7b16ed9d1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9j76s\" (UID: \"db24ee8c-1786-4098-b19a-f6c7b16ed9d1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:15.600476 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.600445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db24ee8c-1786-4098-b19a-f6c7b16ed9d1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9j76s\" (UID: \"db24ee8c-1786-4098-b19a-f6c7b16ed9d1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:15.605078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.605051 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db24ee8c-1786-4098-b19a-f6c7b16ed9d1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9j76s\" (UID: \"db24ee8c-1786-4098-b19a-f6c7b16ed9d1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:15.672518 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.672423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:15.943809 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:15.943682 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s"] Sep 14 04:49:15.947469 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:15.947441 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb24ee8c_1786_4098_b19a_f6c7b16ed9d1.slice/crio-1834d45a008df0d59698d3118b4ccfbfa0d39fc6221998407d55929a4ff81d45 WatchSource:0}: Error finding container 1834d45a008df0d59698d3118b4ccfbfa0d39fc6221998407d55929a4ff81d45: Status 404 returned error can't find the container with id 1834d45a008df0d59698d3118b4ccfbfa0d39fc6221998407d55929a4ff81d45 Sep 14 04:49:16.159544 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:16.159509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" event={"ID":"db24ee8c-1786-4098-b19a-f6c7b16ed9d1","Type":"ContainerStarted","Data":"1834d45a008df0d59698d3118b4ccfbfa0d39fc6221998407d55929a4ff81d45"} Sep 14 04:49:16.910593 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:16.910559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:16.913220 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:16.913193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0273b0e-9540-4ef2-adee-fc134683dcd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdq5h\" (UID: \"c0273b0e-9540-4ef2-adee-fc134683dcd3\") " pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:17.143454 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:17.143406 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdq5h" Sep 14 04:49:17.456983 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:17.456903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdq5h"] Sep 14 04:49:17.461335 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:17.461300 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0273b0e_9540_4ef2_adee_fc134683dcd3.slice/crio-5907cde79a1b82ceb48757a1098f67d7fba00635ca34d94d062ac2a1394d58ac WatchSource:0}: Error finding container 5907cde79a1b82ceb48757a1098f67d7fba00635ca34d94d062ac2a1394d58ac: Status 404 returned error can't find the container with id 5907cde79a1b82ceb48757a1098f67d7fba00635ca34d94d062ac2a1394d58ac Sep 14 04:49:17.546775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:17.546736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:49:18.167961 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.167919 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" event={"ID":"db24ee8c-1786-4098-b19a-f6c7b16ed9d1","Type":"ContainerStarted","Data":"22ec80d7ad21137d74b827191e1042706946dd3f490fe9f51ce0c8f873df1055"} Sep 14 04:49:18.168450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.168101 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:18.169685 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.169647 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdq5h" event={"ID":"c0273b0e-9540-4ef2-adee-fc134683dcd3","Type":"ContainerStarted","Data":"5efad29dcee8e46b5eb2e611c8b33bdc4882d28ab575d4f07204acaf5b409f9c"} Sep 14 04:49:18.169800 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.169699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdq5h" event={"ID":"c0273b0e-9540-4ef2-adee-fc134683dcd3","Type":"ContainerStarted","Data":"5907cde79a1b82ceb48757a1098f67d7fba00635ca34d94d062ac2a1394d58ac"} Sep 14 04:49:18.174005 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.173980 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" Sep 14 04:49:18.183287 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:18.183233 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9j76s" podStartSLOduration=1.8938394779999999 podStartE2EDuration="3.18321596s" totalImagesPullingTime="1.289376482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:18.182658482 +0000 UTC m=+169.140275221" watchObservedRunningTime="2026-09-14 04:49:18.18321596 +0000 UTC m=+169.140832697" Sep 14 04:49:19.175147 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:19.175098 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdq5h" event={"ID":"c0273b0e-9540-4ef2-adee-fc134683dcd3","Type":"ContainerStarted","Data":"90d37a8e2940f5836a2526cc1a42dfc6fb150ca360ab772841dff6af44d1f953"} Sep 14 04:49:20.180194 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:20.180153 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdq5h" event={"ID":"c0273b0e-9540-4ef2-adee-fc134683dcd3","Type":"ContainerStarted","Data":"9c6ae28184e4cc1631b012316582f18c3e73cbe12c24b83e1b2031dcc3a3dcff"} Sep 14 04:49:20.201598 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:20.201497 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hdq5h" podStartSLOduration=32.84046832 podStartE2EDuration="35.20148289s" totalImagesPullingTime="2.36101457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:20.200167104 +0000 UTC m=+171.157783844" watchObservedRunningTime="2026-09-14 04:49:20.20148289 +0000 UTC m=+171.159099626" Sep 14 04:49:21.546933 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:21.546897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:21.550513 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:21.550489 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgrqm\"" Sep 14 04:49:21.558246 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:21.558226 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:21.833877 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:21.833786 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:49:21.837768 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:21.837731 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908c94ac_b68f_4179_9290_123bdc9a1cc1.slice/crio-0e253ca2e5b407d2c2890c51ee109c5ee2efdcf5efaefdb35f8256e3b23d653e WatchSource:0}: Error finding container 0e253ca2e5b407d2c2890c51ee109c5ee2efdcf5efaefdb35f8256e3b23d653e: Status 404 returned error can't find the container with id 0e253ca2e5b407d2c2890c51ee109c5ee2efdcf5efaefdb35f8256e3b23d653e Sep 14 04:49:22.189038 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:22.188999 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" event={"ID":"908c94ac-b68f-4179-9290-123bdc9a1cc1","Type":"ContainerStarted","Data":"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1"} Sep 14 04:49:22.189258 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:22.189057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" event={"ID":"908c94ac-b68f-4179-9290-123bdc9a1cc1","Type":"ContainerStarted","Data":"0e253ca2e5b407d2c2890c51ee109c5ee2efdcf5efaefdb35f8256e3b23d653e"} Sep 14 04:49:22.189258 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:22.189083 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:22.208559 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:22.208505 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" podStartSLOduration=171.208489301 podStartE2EDuration="2m51.208489301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:22.208217824 +0000 UTC m=+173.165834561" watchObservedRunningTime="2026-09-14 04:49:22.208489301 +0000 UTC m=+173.166106038" Sep 14 04:49:23.844599 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.844561 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr"] Sep 14 04:49:23.849244 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.849218 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:23.851729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.851686 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hd2rc\"" Sep 14 04:49:23.851865 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.851698 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 14 04:49:23.852203 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.851993 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 04:49:23.855489 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.852534 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 04:49:23.855489 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.852668 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 14 04:49:23.855489 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.853565 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 04:49:23.864750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.864721 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr"] Sep 14 04:49:23.876953 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.876919 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz"] Sep 14 04:49:23.880160 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.880138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.882315 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.882293 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 14 04:49:23.882488 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.882471 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 14 04:49:23.882555 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.882505 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 14 04:49:23.882605 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.882562 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-djqtm\"" Sep 14 04:49:23.903932 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.903907 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz"] Sep 14 04:49:23.913327 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.913301 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9fmds"] Sep 14 04:49:23.916631 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.916611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:23.918660 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.918633 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vr67q\"" Sep 14 04:49:23.918660 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.918653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 04:49:23.918820 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.918705 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 04:49:23.918820 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.918812 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 04:49:23.965581 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965542 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:23.965761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.965761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c180f68-2b72-4976-9254-4b55371887e3-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.965761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkkf\" (UniqueName: \"kubernetes.io/projected/2c180f68-2b72-4976-9254-4b55371887e3-kube-api-access-7xkkf\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.965761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9379c23-2544-41fd-83dc-63896ef21f78-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:23.965761 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.965998 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:23.965998 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsgj\" (UniqueName: \"kubernetes.io/projected/f9379c23-2544-41fd-83dc-63896ef21f78-kube-api-access-ffsgj\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:23.965998 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:23.965998 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:23.965885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.066481 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9379c23-2544-41fd-83dc-63896ef21f78-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.066481 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.066717 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.066717 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:24.066623 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 14 04:49:24.066717 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsgj\" (UniqueName: \"kubernetes.io/projected/f9379c23-2544-41fd-83dc-63896ef21f78-kube-api-access-ffsgj\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.066717 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:24.066701 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls podName:2c180f68-2b72-4976-9254-4b55371887e3 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:24.566677989 +0000 UTC m=+175.524294724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-6wdcz" (UID: "2c180f68-2b72-4976-9254-4b55371887e3") : secret "kube-state-metrics-tls" not found Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-sys\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/3f4f03ef-d501-4fa7-8643-9e04cea45767-kube-api-access-zzppr\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.066947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.066982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-metrics-client-ca\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c180f68-2b72-4976-9254-4b55371887e3-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-textfile\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-tls\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-wtmp\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkkf\" (UniqueName: \"kubernetes.io/projected/2c180f68-2b72-4976-9254-4b55371887e3-kube-api-access-7xkkf\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067203 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-root\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.067276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9379c23-2544-41fd-83dc-63896ef21f78-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.067765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067658 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c180f68-2b72-4976-9254-4b55371887e3-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.067765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.067765 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.067731 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c180f68-2b72-4976-9254-4b55371887e3-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.070091 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.069908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.070091 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.069908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9379c23-2544-41fd-83dc-63896ef21f78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.070318 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.070292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.081085 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.081063 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsgj\" (UniqueName: \"kubernetes.io/projected/f9379c23-2544-41fd-83dc-63896ef21f78-kube-api-access-ffsgj\") pod \"openshift-state-metrics-67dcf8c979-x2zjr\" (UID: \"f9379c23-2544-41fd-83dc-63896ef21f78\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.081223 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.081107 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkkf\" (UniqueName: \"kubernetes.io/projected/2c180f68-2b72-4976-9254-4b55371887e3-kube-api-access-7xkkf\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.157857 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.157828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-479w5" Sep 14 04:49:24.161066 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.161045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" Sep 14 04:49:24.168361 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168462 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-textfile\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168462 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-tls\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168462 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-wtmp\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168668 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-root\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168668 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-wtmp\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-sys\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168726 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-textfile\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168739 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-sys\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.168768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f4f03ef-d501-4fa7-8643-9e04cea45767-root\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.169047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/3f4f03ef-d501-4fa7-8643-9e04cea45767-kube-api-access-zzppr\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.169047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.169047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.168921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-metrics-client-ca\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.169236 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.169062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.169460 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.169439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f4f03ef-d501-4fa7-8643-9e04cea45767-metrics-client-ca\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.171091 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.171064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.171374 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.171357 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f4f03ef-d501-4fa7-8643-9e04cea45767-node-exporter-tls\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.179180 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.179153 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/3f4f03ef-d501-4fa7-8643-9e04cea45767-kube-api-access-zzppr\") pod \"node-exporter-9fmds\" (UID: \"3f4f03ef-d501-4fa7-8643-9e04cea45767\") " pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.225317 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.225294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9fmds" Sep 14 04:49:24.237531 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:24.237501 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4f03ef_d501_4fa7_8643_9e04cea45767.slice/crio-e7cc5c33b00bc709e00d95acb3dfa2d8db91533d0707a69e2125fba9d0ee689e WatchSource:0}: Error finding container e7cc5c33b00bc709e00d95acb3dfa2d8db91533d0707a69e2125fba9d0ee689e: Status 404 returned error can't find the container with id e7cc5c33b00bc709e00d95acb3dfa2d8db91533d0707a69e2125fba9d0ee689e Sep 14 04:49:24.436333 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.436258 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr"] Sep 14 04:49:24.439592 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:24.439566 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9379c23_2544_41fd_83dc_63896ef21f78.slice/crio-8c93d31105b19e1ab293d855886fe4a16936a905360826ab777860f54ce97bec WatchSource:0}: Error finding container 8c93d31105b19e1ab293d855886fe4a16936a905360826ab777860f54ce97bec: Status 404 returned error can't find the container with id 8c93d31105b19e1ab293d855886fe4a16936a905360826ab777860f54ce97bec Sep 14 04:49:24.573165 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.573137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.575498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.575480 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c180f68-2b72-4976-9254-4b55371887e3-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-6wdcz\" (UID: \"2c180f68-2b72-4976-9254-4b55371887e3\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.789172 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.789070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" Sep 14 04:49:24.928142 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.928096 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:49:24.932172 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.932105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:24.934266 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934225 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 04:49:24.934427 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934292 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934509 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6zk2s\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934600 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934645 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934674 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934610 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 04:49:24.934823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934793 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 04:49:24.935209 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.934964 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 04:49:24.940711 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.940688 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 04:49:24.952844 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:24.952820 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:49:25.074028 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.073956 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz"] Sep 14 04:49:25.077777 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.077897 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.077897 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvgr\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.077897 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.077897 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.077994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.078034 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078152 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.078056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.078147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.078455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.078208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.170146 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:25.170094 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c180f68_2b72_4976_9254_4b55371887e3.slice/crio-8418934b3e364b23122ef2aaf72422c63073f2de106f13332d4b745fb50ce8fb WatchSource:0}: Error finding container 8418934b3e364b23122ef2aaf72422c63073f2de106f13332d4b745fb50ce8fb: Status 404 returned error can't find the container with id 8418934b3e364b23122ef2aaf72422c63073f2de106f13332d4b745fb50ce8fb Sep 14 04:49:25.179190 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179282 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179282 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvgr\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179498 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179851 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.179686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.179905 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:25.179857 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 14 04:49:25.179958 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:25.179917 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls podName:3fad8dd0-79a1-44b9-8452-25ac98c012d1 nodeName:}" failed. No retries permitted until 2026-09-14 04:49:25.679896846 +0000 UTC m=+176.637513572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1") : secret "alertmanager-main-tls" not found Sep 14 04:49:25.180809 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.180763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.181468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.181445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.181677 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.181653 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.182553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.182203 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.182553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.182282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.182775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.182755 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.182942 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.182918 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.183802 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.183780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.184249 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.184226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.184328 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.184202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.184376 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.184340 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.188590 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.188572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvgr\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.200669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.200638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fmds" event={"ID":"3f4f03ef-d501-4fa7-8643-9e04cea45767","Type":"ContainerStarted","Data":"e7cc5c33b00bc709e00d95acb3dfa2d8db91533d0707a69e2125fba9d0ee689e"} Sep 14 04:49:25.207065 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.207037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" event={"ID":"f9379c23-2544-41fd-83dc-63896ef21f78","Type":"ContainerStarted","Data":"42dd06b6be3c8eeaaeea9d9694e18f492fb56c70b6f3225bfe58a1cd874dc5aa"} Sep 14 04:49:25.207200 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.207141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" event={"ID":"f9379c23-2544-41fd-83dc-63896ef21f78","Type":"ContainerStarted","Data":"4adba94ba285e693bcb15acf6500ebc4dca9a909903edd802f685511e3114e2e"} Sep 14 04:49:25.207200 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.207173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" event={"ID":"f9379c23-2544-41fd-83dc-63896ef21f78","Type":"ContainerStarted","Data":"8c93d31105b19e1ab293d855886fe4a16936a905360826ab777860f54ce97bec"} Sep 14 04:49:25.208194 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.208172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" event={"ID":"2c180f68-2b72-4976-9254-4b55371887e3","Type":"ContainerStarted","Data":"8418934b3e364b23122ef2aaf72422c63073f2de106f13332d4b745fb50ce8fb"} Sep 14 04:49:25.683455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.683418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.686049 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.686027 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:25.844396 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:25.844363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:49:26.139592 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.139559 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:49:26.143246 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:26.143218 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fad8dd0_79a1_44b9_8452_25ac98c012d1.slice/crio-dbb2aa3945bd18c1608842d840392493a67f45576d48c3ce30231ca6e223dc9d WatchSource:0}: Error finding container dbb2aa3945bd18c1608842d840392493a67f45576d48c3ce30231ca6e223dc9d: Status 404 returned error can't find the container with id dbb2aa3945bd18c1608842d840392493a67f45576d48c3ce30231ca6e223dc9d Sep 14 04:49:26.213479 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.213443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"dbb2aa3945bd18c1608842d840392493a67f45576d48c3ce30231ca6e223dc9d"} Sep 14 04:49:26.215473 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.215441 2581 generic.go:356] "Generic (PLEG): container finished" podID="3f4f03ef-d501-4fa7-8643-9e04cea45767" containerID="2de241f221b6341f655c7c0b4a80f8280473489a8d7803f00b6b30c195cf11dd" exitCode=0 Sep 14 04:49:26.215597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.215536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fmds" event={"ID":"3f4f03ef-d501-4fa7-8643-9e04cea45767","Type":"ContainerDied","Data":"2de241f221b6341f655c7c0b4a80f8280473489a8d7803f00b6b30c195cf11dd"} Sep 14 04:49:26.218404 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.218367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" event={"ID":"f9379c23-2544-41fd-83dc-63896ef21f78","Type":"ContainerStarted","Data":"e4dcd24a6b2ecd2554bab02d067399eb1a9679a518d05bd6203c26edfeedc940"} Sep 14 04:49:26.249079 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.249037 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-x2zjr" podStartSLOduration=2.03441126 podStartE2EDuration="3.24902072s" totalImagesPullingTime="1.21460946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:26.248462959 +0000 UTC m=+177.206079697" watchObservedRunningTime="2026-09-14 04:49:26.24902072 +0000 UTC m=+177.206637453" Sep 14 04:49:26.745457 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.745420 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-26rmh"] Sep 14 04:49:26.750242 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.750159 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:26.755495 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.754419 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 04:49:26.755495 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.754709 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nvkk5\"" Sep 14 04:49:26.755495 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.754726 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 04:49:26.763144 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.763056 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-26rmh"] Sep 14 04:49:26.896575 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.896544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64l54\" (UniqueName: \"kubernetes.io/projected/b32376a7-aaf0-4d64-a110-73fdc633589d-kube-api-access-64l54\") pod \"downloads-76f99584f4-26rmh\" (UID: \"b32376a7-aaf0-4d64-a110-73fdc633589d\") " pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:26.997096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:26.997049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l54\" (UniqueName: \"kubernetes.io/projected/b32376a7-aaf0-4d64-a110-73fdc633589d-kube-api-access-64l54\") pod \"downloads-76f99584f4-26rmh\" (UID: \"b32376a7-aaf0-4d64-a110-73fdc633589d\") " pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:27.014504 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.014482 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l54\" (UniqueName: \"kubernetes.io/projected/b32376a7-aaf0-4d64-a110-73fdc633589d-kube-api-access-64l54\") pod \"downloads-76f99584f4-26rmh\" (UID: \"b32376a7-aaf0-4d64-a110-73fdc633589d\") " pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:27.066029 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.065941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:27.225910 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.225875 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" event={"ID":"2c180f68-2b72-4976-9254-4b55371887e3","Type":"ContainerStarted","Data":"5d4ecd690ab7600085fdce6bca22aaf8d26b4d4a3b160dc9c613c7c98b0f47bb"} Sep 14 04:49:27.226376 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.225941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" event={"ID":"2c180f68-2b72-4976-9254-4b55371887e3","Type":"ContainerStarted","Data":"ec4882576b41a63b0649b42b1cce57d0db25d2e5d751d1123b57fcbaee94c0d5"} Sep 14 04:49:27.226376 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.225965 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" event={"ID":"2c180f68-2b72-4976-9254-4b55371887e3","Type":"ContainerStarted","Data":"ac192a5fad72e4db739c99a22546b143a8cb5479ffc3644419ed3c1710d888ca"} Sep 14 04:49:27.228233 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.228198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fmds" event={"ID":"3f4f03ef-d501-4fa7-8643-9e04cea45767","Type":"ContainerStarted","Data":"bf2a5cf910a4876a8ffdb6a14288240041142fc66d2c719094640ee9194c5fab"} Sep 14 04:49:27.228431 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.228261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fmds" event={"ID":"3f4f03ef-d501-4fa7-8643-9e04cea45767","Type":"ContainerStarted","Data":"e515e14a9e877804551a82cb05e52a3b1883df5718f3c56cea3ea9e21735a58b"} Sep 14 04:49:27.244703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.244663 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-6wdcz" podStartSLOduration=2.813002358 podStartE2EDuration="4.244646737s" totalImagesPullingTime="1.431644379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:27.243505023 +0000 UTC m=+178.201121760" watchObservedRunningTime="2026-09-14 04:49:27.244646737 +0000 UTC m=+178.202263473" Sep 14 04:49:27.262049 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.262001 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9fmds" podStartSLOduration=3.222525736 podStartE2EDuration="4.261985083s" totalImagesPullingTime="977.457785ms" totalInitContainerRuntime="62.001562ms" isStatefulPod=true podCreationTimestamp="2026-09-14 04:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:27.261237392 +0000 UTC m=+178.218854119" watchObservedRunningTime="2026-09-14 04:49:27.261985083 +0000 UTC m=+178.219601819" Sep 14 04:49:27.517376 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:27.517350 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-26rmh"] Sep 14 04:49:27.518866 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:27.518833 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32376a7_aaf0_4d64_a110_73fdc633589d.slice/crio-2cd3484f51fac86d5052fbe3c00aeb3d150387ab198f6b7793b99e0131b11511 WatchSource:0}: Error finding container 2cd3484f51fac86d5052fbe3c00aeb3d150387ab198f6b7793b99e0131b11511: Status 404 returned error can't find the container with id 2cd3484f51fac86d5052fbe3c00aeb3d150387ab198f6b7793b99e0131b11511 Sep 14 04:49:28.165360 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.165322 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77d5df6575-79nvs"] Sep 14 04:49:28.168916 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.168892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.171428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.171404 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 04:49:28.172441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.171780 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 04:49:28.172441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.171840 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-97qsz\"" Sep 14 04:49:28.172441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.172172 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 04:49:28.172441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.172302 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bebdt19n9cihu\"" Sep 14 04:49:28.172441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.172351 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 04:49:28.184334 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.184312 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77d5df6575-79nvs"] Sep 14 04:49:28.233697 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.233661 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-26rmh" event={"ID":"b32376a7-aaf0-4d64-a110-73fdc633589d","Type":"ContainerStarted","Data":"2cd3484f51fac86d5052fbe3c00aeb3d150387ab198f6b7793b99e0131b11511"} Sep 14 04:49:28.235626 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.235600 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e" exitCode=0 Sep 14 04:49:28.235766 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.235660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e"} Sep 14 04:49:28.309312 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.309275 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-client-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.309779 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.309478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chn8\" (UniqueName: \"kubernetes.io/projected/b46b88f2-c4f2-4169-ac2a-1a91d773d637-kube-api-access-9chn8\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.309779 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.309532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.310006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.309787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-tls\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.310006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.309879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-metrics-server-audit-profiles\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.310142 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.310092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b46b88f2-c4f2-4169-ac2a-1a91d773d637-audit-log\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.310212 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.310168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-client-certs\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.410970 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.410935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411153 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-tls\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411153 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-metrics-server-audit-profiles\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411266 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b46b88f2-c4f2-4169-ac2a-1a91d773d637-audit-log\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411266 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-client-certs\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411266 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-client-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411417 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chn8\" (UniqueName: \"kubernetes.io/projected/b46b88f2-c4f2-4169-ac2a-1a91d773d637-kube-api-access-9chn8\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411618 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411586 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b46b88f2-c4f2-4169-ac2a-1a91d773d637-audit-log\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.411894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.411848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.412366 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.412321 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b46b88f2-c4f2-4169-ac2a-1a91d773d637-metrics-server-audit-profiles\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.413817 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.413797 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-client-certs\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.414232 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.414210 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-client-ca-bundle\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.414411 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.414392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b46b88f2-c4f2-4169-ac2a-1a91d773d637-secret-metrics-server-tls\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.420327 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.420276 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chn8\" (UniqueName: \"kubernetes.io/projected/b46b88f2-c4f2-4169-ac2a-1a91d773d637-kube-api-access-9chn8\") pod \"metrics-server-77d5df6575-79nvs\" (UID: \"b46b88f2-c4f2-4169-ac2a-1a91d773d637\") " pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.480359 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.480328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:28.762820 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:28.762731 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77d5df6575-79nvs"] Sep 14 04:49:28.766434 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:28.766399 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46b88f2_c4f2_4169_ac2a_1a91d773d637.slice/crio-e1c51d1c5b581155e68c48629f0f51543be95c7b33364afae11aad939381d51b WatchSource:0}: Error finding container e1c51d1c5b581155e68c48629f0f51543be95c7b33364afae11aad939381d51b: Status 404 returned error can't find the container with id e1c51d1c5b581155e68c48629f0f51543be95c7b33364afae11aad939381d51b Sep 14 04:49:29.241652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:29.241613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" event={"ID":"b46b88f2-c4f2-4169-ac2a-1a91d773d637","Type":"ContainerStarted","Data":"e1c51d1c5b581155e68c48629f0f51543be95c7b33364afae11aad939381d51b"} Sep 14 04:49:30.128222 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.128105 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:49:30.133444 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.133273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.135722 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.135389 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mldrm\"" Sep 14 04:49:30.135722 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.135658 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 04:49:30.135896 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.135858 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 04:49:30.136417 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.136255 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 04:49:30.136553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.136433 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 04:49:30.138696 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.138388 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 04:49:30.138696 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.138518 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 04:49:30.138696 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.138579 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 04:49:30.138916 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.138780 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 04:49:30.138916 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.138799 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 04:49:30.139077 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.139007 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 04:49:30.139440 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.139288 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-69dn3k14cp6iv\"" Sep 14 04:49:30.139440 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.139297 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 04:49:30.139581 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.139546 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 04:49:30.140009 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.139752 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 04:49:30.140984 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.140909 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 04:49:30.154293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.154267 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:49:30.228987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.228830 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.228987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.228881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.228987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.228918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.228987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.228946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.228987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.228982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229378 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229471 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229535 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qz6x\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.229852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.229607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.250486 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.250455 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040"} Sep 14 04:49:30.250890 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.250523 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d"} Sep 14 04:49:30.250890 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.250553 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158"} Sep 14 04:49:30.250890 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.250592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873"} Sep 14 04:49:30.330667 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330667 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.330894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.330973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qz6x\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331691 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.331691 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.332981 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.331863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.332981 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.332155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.332981 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.332360 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.333449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.333658 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.334699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.334927 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.335283 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.336070 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.335742 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.337468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.336528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.337468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.336679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.337468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.337042 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.337468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.337191 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.337468 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.337278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.338321 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.338278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.338625 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.338603 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.338807 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.338772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.339199 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.339152 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.340751 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.340711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.342341 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.342317 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qz6x\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x\") pod \"prometheus-k8s-0\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.444971 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.444915 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:30.862204 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:30.861813 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:49:30.867741 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:49:30.867708 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe62de_8648_42d9_880f_fef43aadd2a5.slice/crio-7d554c9a14e6395d2b5873f18b181b1454b07e4513e866ed013e69229e2c1aa6 WatchSource:0}: Error finding container 7d554c9a14e6395d2b5873f18b181b1454b07e4513e866ed013e69229e2c1aa6: Status 404 returned error can't find the container with id 7d554c9a14e6395d2b5873f18b181b1454b07e4513e866ed013e69229e2c1aa6 Sep 14 04:49:31.260780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.260683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac"} Sep 14 04:49:31.263215 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.262950 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" event={"ID":"b46b88f2-c4f2-4169-ac2a-1a91d773d637","Type":"ContainerStarted","Data":"39b7ead75edf735131742777dbf48921d6ed87ccf16d92eace0fa3502f178f28"} Sep 14 04:49:31.265647 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.265619 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" exitCode=0 Sep 14 04:49:31.265746 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.265673 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} Sep 14 04:49:31.265746 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.265733 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"7d554c9a14e6395d2b5873f18b181b1454b07e4513e866ed013e69229e2c1aa6"} Sep 14 04:49:31.286037 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:31.285966 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" podStartSLOduration=1.43605058 podStartE2EDuration="3.285945281s" totalImagesPullingTime="1.849894701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:31.283626937 +0000 UTC m=+182.241243674" watchObservedRunningTime="2026-09-14 04:49:31.285945281 +0000 UTC m=+182.243562018" Sep 14 04:49:32.032463 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:32.032044 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:49:32.276481 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:32.276441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerStarted","Data":"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b"} Sep 14 04:49:32.303452 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:32.302962 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.72560077 podStartE2EDuration="8.302941604s" totalImagesPullingTime="3.56229434s" totalInitContainerRuntime="15.046494ms" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:32.301655873 +0000 UTC m=+183.259272611" watchObservedRunningTime="2026-09-14 04:49:32.302941604 +0000 UTC m=+183.260558342" Sep 14 04:49:35.297278 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:35.297228 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} Sep 14 04:49:35.297722 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:35.297331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} Sep 14 04:49:37.313090 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:37.313060 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} Sep 14 04:49:37.313520 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:37.313163 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} Sep 14 04:49:37.313520 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:37.313249 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} Sep 14 04:49:38.322233 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:38.322193 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerStarted","Data":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} Sep 14 04:49:38.354594 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:38.354543 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.729122591 podStartE2EDuration="8.354525686s" totalImagesPullingTime="5.604126767s" totalInitContainerRuntime="21.276328ms" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:38.352382078 +0000 UTC m=+189.309998838" watchObservedRunningTime="2026-09-14 04:49:38.354525686 +0000 UTC m=+189.312142422" Sep 14 04:49:40.445533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:40.445491 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:49:42.038179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:42.038146 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:45.354475 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:45.354430 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-26rmh" event={"ID":"b32376a7-aaf0-4d64-a110-73fdc633589d","Type":"ContainerStarted","Data":"9d0597be126882defa5d20a23adb9c642e17335c103463df84c9066e1fd59dd5"} Sep 14 04:49:45.354951 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:45.354615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:45.371471 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:45.371398 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-26rmh" Sep 14 04:49:45.373571 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:45.373515 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-26rmh" podStartSLOduration=2.458181927 podStartE2EDuration="19.373496255s" totalImagesPullingTime="16.915314328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:45.371341425 +0000 UTC m=+196.328958158" watchObservedRunningTime="2026-09-14 04:49:45.373496255 +0000 UTC m=+196.331112993" Sep 14 04:49:48.480631 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:48.480592 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:48.481202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:48.480669 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:49:50.380815 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:50.380769 2581 generic.go:356] "Generic (PLEG): container finished" podID="e3b45323-6c9b-4747-8665-d81e85a37840" containerID="446820d03d612232cebbd0c5a75f58cc223fb331438b09267d49fb4cd740f7d4" exitCode=0 Sep 14 04:49:50.381295 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:50.380850 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-pkk55" event={"ID":"e3b45323-6c9b-4747-8665-d81e85a37840","Type":"ContainerDied","Data":"446820d03d612232cebbd0c5a75f58cc223fb331438b09267d49fb4cd740f7d4"} Sep 14 04:49:50.381370 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:50.381306 2581 scope.go:117] "RemoveContainer" containerID="446820d03d612232cebbd0c5a75f58cc223fb331438b09267d49fb4cd740f7d4" Sep 14 04:49:51.388576 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:51.388531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-pkk55" event={"ID":"e3b45323-6c9b-4747-8665-d81e85a37840","Type":"ContainerStarted","Data":"c0f905803602d75beeaccfb38bef37b6d962a987cbb03a526a997c7a2f19de9d"} Sep 14 04:49:55.406629 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:55.406587 2581 generic.go:356] "Generic (PLEG): container finished" podID="aa28282c-eebb-4e68-a437-97c35d9c3293" containerID="bba80976edc88f87ca3d080178668460b1a800ab0501092ef7f5c78109a0515c" exitCode=0 Sep 14 04:49:55.407043 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:55.406649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" event={"ID":"aa28282c-eebb-4e68-a437-97c35d9c3293","Type":"ContainerDied","Data":"bba80976edc88f87ca3d080178668460b1a800ab0501092ef7f5c78109a0515c"} Sep 14 04:49:55.407043 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:55.406968 2581 scope.go:117] "RemoveContainer" containerID="bba80976edc88f87ca3d080178668460b1a800ab0501092ef7f5c78109a0515c" Sep 14 04:49:56.412646 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:56.412606 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5c25b" event={"ID":"aa28282c-eebb-4e68-a437-97c35d9c3293","Type":"ContainerStarted","Data":"c36fe377489e3f51b4d7fb8cdd96f61b7f7217824d0ae7992d556f6a1fee525c"} Sep 14 04:49:57.058261 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.058217 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" containerName="registry" containerID="cri-o://5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1" gracePeriod=30 Sep 14 04:49:57.339358 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.339276 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:57.416716 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416687 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.416716 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416717 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416756 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416789 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416846 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416894 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416936 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb65n\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.416987 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets\") pod \"908c94ac-b68f-4179-9290-123bdc9a1cc1\" (UID: \"908c94ac-b68f-4179-9290-123bdc9a1cc1\") " Sep 14 04:49:57.417293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.417227 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:49:57.417633 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.417311 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-trusted-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.417841 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.417791 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:49:57.418737 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.418592 2581 generic.go:356] "Generic (PLEG): container finished" podID="908c94ac-b68f-4179-9290-123bdc9a1cc1" containerID="5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1" exitCode=0 Sep 14 04:49:57.418737 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.418646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" event={"ID":"908c94ac-b68f-4179-9290-123bdc9a1cc1","Type":"ContainerDied","Data":"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1"} Sep 14 04:49:57.418737 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.418686 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" Sep 14 04:49:57.418948 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.418696 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db78995bf-vw9bj" event={"ID":"908c94ac-b68f-4179-9290-123bdc9a1cc1","Type":"ContainerDied","Data":"0e253ca2e5b407d2c2890c51ee109c5ee2efdcf5efaefdb35f8256e3b23d653e"} Sep 14 04:49:57.418948 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.418717 2581 scope.go:117] "RemoveContainer" containerID="5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1" Sep 14 04:49:57.419803 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.419776 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:49:57.420105 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.420069 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:49:57.420331 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.420205 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:49:57.420476 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.420394 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n" (OuterVolumeSpecName: "kube-api-access-hb65n") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "kube-api-access-hb65n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:49:57.420655 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.420625 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:49:57.428103 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.428076 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "908c94ac-b68f-4179-9290-123bdc9a1cc1" (UID: "908c94ac-b68f-4179-9290-123bdc9a1cc1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:49:57.437051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.437033 2581 scope.go:117] "RemoveContainer" containerID="5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1" Sep 14 04:49:57.437422 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:49:57.437386 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1\": container with ID starting with 5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1 not found: ID does not exist" containerID="5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1" Sep 14 04:49:57.437486 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.437430 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1"} err="failed to get container status \"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1\": rpc error: code = NotFound desc = could not find container \"5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1\": container with ID starting with 5cbcc1bfb37158353fe9d2c994237cf3597ec113ee016bad335114e2b3552dc1 not found: ID does not exist" Sep 14 04:49:57.518196 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518160 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-installation-pull-secrets\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518196 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518192 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-tls\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518196 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518202 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/908c94ac-b68f-4179-9290-123bdc9a1cc1-image-registry-private-configuration\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518211 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/908c94ac-b68f-4179-9290-123bdc9a1cc1-registry-certificates\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518220 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-bound-sa-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518229 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/908c94ac-b68f-4179-9290-123bdc9a1cc1-ca-trust-extracted\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.518425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.518237 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb65n\" (UniqueName: \"kubernetes.io/projected/908c94ac-b68f-4179-9290-123bdc9a1cc1-kube-api-access-hb65n\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:49:57.734553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.734521 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:49:57.737844 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:57.737813 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6db78995bf-vw9bj"] Sep 14 04:49:59.551667 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:49:59.551625 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" path="/var/lib/kubelet/pods/908c94ac-b68f-4179-9290-123bdc9a1cc1/volumes" Sep 14 04:50:08.485459 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:08.485430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:50:08.489108 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:08.489088 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77d5df6575-79nvs" Sep 14 04:50:30.445390 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:30.445334 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:30.464437 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:30.464403 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:30.564937 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:30.564906 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:41.503325 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:41.503278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:50:41.505589 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:41.505568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c054e08e-84af-4062-bd3c-4456cea1f2e5-metrics-certs\") pod \"network-metrics-daemon-xdm24\" (UID: \"c054e08e-84af-4062-bd3c-4456cea1f2e5\") " pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:50:41.552151 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:41.552110 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgkvn\"" Sep 14 04:50:41.558004 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:41.557986 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdm24" Sep 14 04:50:41.837528 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:41.837446 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xdm24"] Sep 14 04:50:41.841884 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:50:41.841849 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc054e08e_84af_4062_bd3c_4456cea1f2e5.slice/crio-3a0a5538cf39cb190081700853ba45041c3e38ef0a3f7a2c4d5b971e99001272 WatchSource:0}: Error finding container 3a0a5538cf39cb190081700853ba45041c3e38ef0a3f7a2c4d5b971e99001272: Status 404 returned error can't find the container with id 3a0a5538cf39cb190081700853ba45041c3e38ef0a3f7a2c4d5b971e99001272 Sep 14 04:50:42.597542 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:42.597489 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdm24" event={"ID":"c054e08e-84af-4062-bd3c-4456cea1f2e5","Type":"ContainerStarted","Data":"3a0a5538cf39cb190081700853ba45041c3e38ef0a3f7a2c4d5b971e99001272"} Sep 14 04:50:43.602752 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:43.602718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdm24" event={"ID":"c054e08e-84af-4062-bd3c-4456cea1f2e5","Type":"ContainerStarted","Data":"ccd12f9c31fd79d4a745d3d584cfc86631746e8fbb10329617d840f1e9c502b0"} Sep 14 04:50:43.603175 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:43.602773 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdm24" event={"ID":"c054e08e-84af-4062-bd3c-4456cea1f2e5","Type":"ContainerStarted","Data":"727fcbd45ace113ca8a0231234e1bb6668379d481747ddad9171f90b70b8b264"} Sep 14 04:50:43.619630 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:43.619570 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xdm24" podStartSLOduration=252.60862837 podStartE2EDuration="4m13.619547795s" totalImagesPullingTime="1.010919425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:50:43.618271363 +0000 UTC m=+254.575888100" watchObservedRunningTime="2026-09-14 04:50:43.619547795 +0000 UTC m=+254.577164535" Sep 14 04:50:44.262314 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262278 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:44.262622 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262595 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="alertmanager" containerID="cri-o://9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873" gracePeriod=120 Sep 14 04:50:44.262931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262650 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-metric" containerID="cri-o://3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac" gracePeriod=120 Sep 14 04:50:44.262931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262718 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="config-reloader" containerID="cri-o://17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158" gracePeriod=120 Sep 14 04:50:44.262931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262788 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="prom-label-proxy" containerID="cri-o://fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b" gracePeriod=120 Sep 14 04:50:44.262931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262684 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-web" containerID="cri-o://922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d" gracePeriod=120 Sep 14 04:50:44.262931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.262716 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy" containerID="cri-o://95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040" gracePeriod=120 Sep 14 04:50:44.612293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612260 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b" exitCode=0 Sep 14 04:50:44.612293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612284 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac" exitCode=0 Sep 14 04:50:44.612293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612291 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040" exitCode=0 Sep 14 04:50:44.612293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612296 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158" exitCode=0 Sep 14 04:50:44.612293 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612302 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873" exitCode=0 Sep 14 04:50:44.612780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b"} Sep 14 04:50:44.612780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612390 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac"} Sep 14 04:50:44.612780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040"} Sep 14 04:50:44.612780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158"} Sep 14 04:50:44.612780 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:44.612465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873"} Sep 14 04:50:45.509989 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.509963 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:45.619827 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.619797 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerID="922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d" exitCode=0 Sep 14 04:50:45.620201 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.619853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d"} Sep 14 04:50:45.620201 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.619875 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:45.620201 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.619909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fad8dd0-79a1-44b9-8452-25ac98c012d1","Type":"ContainerDied","Data":"dbb2aa3945bd18c1608842d840392493a67f45576d48c3ce30231ca6e223dc9d"} Sep 14 04:50:45.620201 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.619942 2581 scope.go:117] "RemoveContainer" containerID="fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b" Sep 14 04:50:45.627775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.627754 2581 scope.go:117] "RemoveContainer" containerID="3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac" Sep 14 04:50:45.634616 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.634597 2581 scope.go:117] "RemoveContainer" containerID="95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040" Sep 14 04:50:45.636686 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636668 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.636775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636732 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.636775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636764 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.636883 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636793 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.636883 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636831 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.636883 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636860 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636888 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtvgr\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636915 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636964 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.636991 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637041 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637068 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637094 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets\") pod \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\" (UID: \"3fad8dd0-79a1-44b9-8452-25ac98c012d1\") " Sep 14 04:50:45.637375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637209 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:45.637375 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637266 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:45.637528 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637432 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.637585 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637538 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fad8dd0-79a1-44b9-8452-25ac98c012d1-metrics-client-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.637639 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.637583 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:50:45.640483 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.640429 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out" (OuterVolumeSpecName: "config-out") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:50:45.640597 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.640481 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.640667 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.640629 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.640944 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.640919 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr" (OuterVolumeSpecName: "kube-api-access-rtvgr") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "kube-api-access-rtvgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:50:45.641033 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.640934 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.641366 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.641331 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.641972 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.641942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:50:45.642484 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.642464 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume" (OuterVolumeSpecName: "config-volume") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.644532 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.644470 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.644622 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.644607 2581 scope.go:117] "RemoveContainer" containerID="922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d" Sep 14 04:50:45.651572 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.651551 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config" (OuterVolumeSpecName: "web-config") pod "3fad8dd0-79a1-44b9-8452-25ac98c012d1" (UID: "3fad8dd0-79a1-44b9-8452-25ac98c012d1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:45.659914 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.659896 2581 scope.go:117] "RemoveContainer" containerID="17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158" Sep 14 04:50:45.667175 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.667151 2581 scope.go:117] "RemoveContainer" containerID="9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873" Sep 14 04:50:45.674440 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.674418 2581 scope.go:117] "RemoveContainer" containerID="5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e" Sep 14 04:50:45.682145 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682107 2581 scope.go:117] "RemoveContainer" containerID="fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b" Sep 14 04:50:45.682448 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.682427 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b\": container with ID starting with fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b not found: ID does not exist" containerID="fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b" Sep 14 04:50:45.682499 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682458 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b"} err="failed to get container status \"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b\": rpc error: code = NotFound desc = could not find container \"fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b\": container with ID starting with fb98d0b4772e1968216720e72271febc244a4345206aa0c4f7e3a75d18e68b9b not found: ID does not exist" Sep 14 04:50:45.682499 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682477 2581 scope.go:117] "RemoveContainer" containerID="3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac" Sep 14 04:50:45.682739 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.682722 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac\": container with ID starting with 3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac not found: ID does not exist" containerID="3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac" Sep 14 04:50:45.682790 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682746 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac"} err="failed to get container status \"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac\": rpc error: code = NotFound desc = could not find container \"3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac\": container with ID starting with 3635209b3ad07857787ad7584d390b0276b65b70671266c3172afccf1e1906ac not found: ID does not exist" Sep 14 04:50:45.682790 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682762 2581 scope.go:117] "RemoveContainer" containerID="95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040" Sep 14 04:50:45.682981 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.682967 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040\": container with ID starting with 95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040 not found: ID does not exist" containerID="95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040" Sep 14 04:50:45.683024 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040"} err="failed to get container status \"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040\": rpc error: code = NotFound desc = could not find container \"95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040\": container with ID starting with 95687a3810909963b5f1d8f91779bf45948f48a813432f079c16be958030b040 not found: ID does not exist" Sep 14 04:50:45.683024 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.682995 2581 scope.go:117] "RemoveContainer" containerID="922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d" Sep 14 04:50:45.683239 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.683223 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d\": container with ID starting with 922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d not found: ID does not exist" containerID="922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d" Sep 14 04:50:45.683289 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683244 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d"} err="failed to get container status \"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d\": rpc error: code = NotFound desc = could not find container \"922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d\": container with ID starting with 922124beedb54d42d8a3db8f05342e0bc7fa414fbf417787f80f695c688c7b3d not found: ID does not exist" Sep 14 04:50:45.683289 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683259 2581 scope.go:117] "RemoveContainer" containerID="17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158" Sep 14 04:50:45.683470 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.683456 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158\": container with ID starting with 17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158 not found: ID does not exist" containerID="17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158" Sep 14 04:50:45.683509 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683472 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158"} err="failed to get container status \"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158\": rpc error: code = NotFound desc = could not find container \"17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158\": container with ID starting with 17ec59c7c1516fc418b949b62e62d1ffe055c3fa045f3b71a6a85ef53c22e158 not found: ID does not exist" Sep 14 04:50:45.683509 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683484 2581 scope.go:117] "RemoveContainer" containerID="9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873" Sep 14 04:50:45.683701 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.683685 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873\": container with ID starting with 9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873 not found: ID does not exist" containerID="9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873" Sep 14 04:50:45.683746 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683703 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873"} err="failed to get container status \"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873\": rpc error: code = NotFound desc = could not find container \"9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873\": container with ID starting with 9a5c7b890524ac2e5f308c6aa94f69890012662d962e125cdec435a3e4fd7873 not found: ID does not exist" Sep 14 04:50:45.683746 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683716 2581 scope.go:117] "RemoveContainer" containerID="5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e" Sep 14 04:50:45.683927 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:45.683911 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e\": container with ID starting with 5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e not found: ID does not exist" containerID="5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e" Sep 14 04:50:45.683963 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.683930 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e"} err="failed to get container status \"5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e\": rpc error: code = NotFound desc = could not find container \"5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e\": container with ID starting with 5621d5234ade0b061de25ecbde1309411debd541af4e6cda2025add402def37e not found: ID does not exist" Sep 14 04:50:45.738424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738385 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-out\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738418 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-web-config\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738433 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-tls-assets\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738445 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-cluster-tls-config\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738458 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-main-tls\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738470 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fad8dd0-79a1-44b9-8452-25ac98c012d1-alertmanager-main-db\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738482 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738494 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvgr\" (UniqueName: \"kubernetes.io/projected/3fad8dd0-79a1-44b9-8452-25ac98c012d1-kube-api-access-rtvgr\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738508 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738519 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.738649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.738531 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fad8dd0-79a1-44b9-8452-25ac98c012d1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:45.943792 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.943753 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:45.947743 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.947711 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:45.978852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.978815 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:45.979104 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979093 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-web" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979106 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-web" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979135 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" containerName="registry" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979141 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" containerName="registry" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979147 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="init-config-reloader" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979153 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="init-config-reloader" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979162 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="config-reloader" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979167 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="config-reloader" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979177 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979182 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy" Sep 14 04:50:45.979187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979188 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="prom-label-proxy" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979194 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="prom-label-proxy" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979200 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-metric" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979206 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-metric" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979213 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="alertmanager" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979218 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="alertmanager" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979263 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979273 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="prom-label-proxy" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979283 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="908c94ac-b68f-4179-9290-123bdc9a1cc1" containerName="registry" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979293 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="config-reloader" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979301 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-web" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979309 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="kube-rbac-proxy-metric" Sep 14 04:50:45.979461 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.979314 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" containerName="alertmanager" Sep 14 04:50:45.984438 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.984410 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:45.986469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986443 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 04:50:45.986589 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986542 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 04:50:45.986589 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986578 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 04:50:45.986719 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986645 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 04:50:45.986719 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986695 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 04:50:45.986871 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 04:50:45.986984 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.986910 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 04:50:45.987247 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.987228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 04:50:45.987315 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.987303 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6zk2s\"" Sep 14 04:50:45.992074 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.991963 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 04:50:45.998971 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:45.998944 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:46.141441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141441 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-web-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141676 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-config-out\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcwh\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-kube-api-access-gtcwh\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-config-volume\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.141884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.141775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243295 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243295 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-web-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243295 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-config-out\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcwh\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-kube-api-access-gtcwh\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243428 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-config-volume\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.243641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.243626 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.244202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.244174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246094 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246231 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246214 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246302 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.244181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21e57896-5485-45e4-b4c9-1f0248ef662d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246374 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-web-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246374 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246365 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246471 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246351 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246655 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-config-volume\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.246845 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.246826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.247112 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.247097 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21e57896-5485-45e4-b4c9-1f0248ef662d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.247832 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.247816 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21e57896-5485-45e4-b4c9-1f0248ef662d-config-out\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.254443 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.254423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcwh\" (UniqueName: \"kubernetes.io/projected/21e57896-5485-45e4-b4c9-1f0248ef662d-kube-api-access-gtcwh\") pod \"alertmanager-main-0\" (UID: \"21e57896-5485-45e4-b4c9-1f0248ef662d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.295066 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.295039 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 04:50:46.580226 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.579878 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 04:50:46.582765 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:50:46.582739 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e57896_5485_45e4_b4c9_1f0248ef662d.slice/crio-ec11443cd14ab7cdcf09f39712509416f052116a7bfe990185db0fd4cbcaee55 WatchSource:0}: Error finding container ec11443cd14ab7cdcf09f39712509416f052116a7bfe990185db0fd4cbcaee55: Status 404 returned error can't find the container with id ec11443cd14ab7cdcf09f39712509416f052116a7bfe990185db0fd4cbcaee55 Sep 14 04:50:46.624653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:46.624620 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"ec11443cd14ab7cdcf09f39712509416f052116a7bfe990185db0fd4cbcaee55"} Sep 14 04:50:47.551046 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:47.551011 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fad8dd0-79a1-44b9-8452-25ac98c012d1" path="/var/lib/kubelet/pods/3fad8dd0-79a1-44b9-8452-25ac98c012d1/volumes" Sep 14 04:50:47.631082 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:47.631051 2581 generic.go:356] "Generic (PLEG): container finished" podID="21e57896-5485-45e4-b4c9-1f0248ef662d" containerID="8f07b4abd8afa9d72881c1a7c4b887afa4f2732ef99d7424a8600fc9f7c9d7ea" exitCode=0 Sep 14 04:50:47.631475 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:47.631112 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerDied","Data":"8f07b4abd8afa9d72881c1a7c4b887afa4f2732ef99d7424a8600fc9f7c9d7ea"} Sep 14 04:50:48.639823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639794 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"6e713bca445cd370484b96d7714f2cbdb4681cad57a8d165f0dcf1d96486815c"} Sep 14 04:50:48.640277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"0b02ac1fafa55f178d915156f1d154deebb4e9ec2ed730657b460b961b3bfeb7"} Sep 14 04:50:48.640277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639877 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"2192fb24e7ea1baf5f39685194ca2ab7135a1ab5b4165811eaa9959bcb94a699"} Sep 14 04:50:48.640277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639904 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"01101631c36ebf09cd9cff3b774946b9be8e396194bb10ae2ea8f93dc4679a63"} Sep 14 04:50:48.640277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639927 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"4cfc106b5fcfb32151992f4bd1d122239369f2cd9e3ae499b8268c85edd25143"} Sep 14 04:50:48.640277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.639948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21e57896-5485-45e4-b4c9-1f0248ef662d","Type":"ContainerStarted","Data":"840141d4d356dbc2a50bf086b3915dce72d6c396d5b6555684f7a4b6c48a6b0a"} Sep 14 04:50:48.668886 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.668857 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:48.669191 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669158 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="prometheus" containerID="cri-o://325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" gracePeriod=600 Sep 14 04:50:48.669346 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669189 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="config-reloader" containerID="cri-o://762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" gracePeriod=600 Sep 14 04:50:48.669346 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669197 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="thanos-sidecar" containerID="cri-o://95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" gracePeriod=600 Sep 14 04:50:48.669346 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669166 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" gracePeriod=600 Sep 14 04:50:48.669346 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669188 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-web" containerID="cri-o://cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" gracePeriod=600 Sep 14 04:50:48.669346 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.669191 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy" containerID="cri-o://f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" gracePeriod=600 Sep 14 04:50:48.674315 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.674255 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.654784152 podStartE2EDuration="3.674237354s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.453202ms" isStatefulPod=false podCreationTimestamp="2026-09-14 04:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:50:48.6718971 +0000 UTC m=+259.629513837" watchObservedRunningTime="2026-09-14 04:50:48.674237354 +0000 UTC m=+259.631854093" Sep 14 04:50:48.899076 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:48.899026 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.066335 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066304 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066480 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066342 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066480 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066376 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066480 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066399 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066512 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066565 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066596 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066626 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066651 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.066679 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066677 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066700 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qz6x\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066723 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066753 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066779 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066794 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066821 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066875 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066939 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.066975 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067006 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067012 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067621 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067037 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1\") pod \"f2fe62de-8648-42d9-880f-fef43aadd2a5\" (UID: \"f2fe62de-8648-42d9-880f-fef43aadd2a5\") " Sep 14 04:50:49.067621 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067301 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-metrics-client-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.067731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067629 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.067980 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067956 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.068051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.067954 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:50:49.069202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.068936 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.069202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.069034 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.069202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.069168 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.069415 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.069306 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.069666 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.069629 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.069666 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.069643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.070047 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.070015 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.070345 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.070305 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.070618 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.070592 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:50:49.070729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.070691 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.070851 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.070732 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x" (OuterVolumeSpecName: "kube-api-access-4qz6x") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "kube-api-access-4qz6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:50:49.071816 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.071786 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:50:49.072213 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.072188 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out" (OuterVolumeSpecName: "config-out") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:50:49.072351 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.072332 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config" (OuterVolumeSpecName: "config") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.072442 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.072404 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.079867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.079848 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config" (OuterVolumeSpecName: "web-config") pod "f2fe62de-8648-42d9-880f-fef43aadd2a5" (UID: "f2fe62de-8648-42d9-880f-fef43aadd2a5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167690 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167722 2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-kube-rbac-proxy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167733 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167743 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-db\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167751 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167750 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167760 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-tls-assets\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167768 2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-config\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167776 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qz6x\" (UniqueName: \"kubernetes.io/projected/f2fe62de-8648-42d9-880f-fef43aadd2a5-kube-api-access-4qz6x\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167784 2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-grpc-tls\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167792 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167800 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167809 2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-secret-metrics-client-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167818 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167826 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167834 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167844 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fe62de-8648-42d9-880f-fef43aadd2a5-config-out\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167851 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2fe62de-8648-42d9-880f-fef43aadd2a5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167859 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-web-config\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.167973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.167868 2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2fe62de-8648-42d9-880f-fef43aadd2a5-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:50:49.647096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647064 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" exitCode=0 Sep 14 04:50:49.647096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647090 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" exitCode=0 Sep 14 04:50:49.647096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647097 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" exitCode=0 Sep 14 04:50:49.647096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647103 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" exitCode=0 Sep 14 04:50:49.647096 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647108 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" exitCode=0 Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647126 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" exitCode=0 Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647158 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647179 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647228 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647256 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647283 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647290 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} Sep 14 04:50:49.647669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.647355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2fe62de-8648-42d9-880f-fef43aadd2a5","Type":"ContainerDied","Data":"7d554c9a14e6395d2b5873f18b181b1454b07e4513e866ed013e69229e2c1aa6"} Sep 14 04:50:49.655013 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.654991 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.662150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.662131 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.669370 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.669349 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.669451 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.669354 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:49.677942 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.677921 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.678419 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.678399 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:49.685143 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.685111 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.692563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.692545 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.699779 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.699756 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.700223 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.700196 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.700311 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700235 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.700311 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700261 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.700559 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.700541 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.700604 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700564 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.700604 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700581 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.700822 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.700807 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.700877 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700832 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.700877 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.700851 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.701087 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.701068 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.701149 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701090 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.701149 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701102 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.701354 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.701337 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.701398 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701358 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.701398 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701372 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.701618 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.701600 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.701657 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701623 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.701657 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701637 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.701880 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:50:49.701863 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.701939 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701886 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.701939 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.701904 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.702193 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702167 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.702243 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702193 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.702443 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702426 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.702443 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702443 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.702657 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702642 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.702657 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702656 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.702881 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702859 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.702881 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.702878 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.703100 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703081 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.703176 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703103 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.703323 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703306 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.703379 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703324 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.703533 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703516 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.703604 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703538 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.703768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703751 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.703827 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703770 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.703971 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703956 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.704010 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.703970 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.704181 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704164 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.704234 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704181 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.704371 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704356 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.704409 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704371 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.704574 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704557 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.704641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704576 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.704778 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704761 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.704828 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704778 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.704970 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704955 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.705021 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.704970 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.705232 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705216 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.705232 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705231 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.705436 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705419 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.705496 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705438 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.705641 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705626 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.705682 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705640 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.705855 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705838 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.705917 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.705856 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.706079 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706062 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.706156 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706080 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.706366 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706349 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.706408 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706365 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.706569 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706553 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.706606 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706569 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.706803 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706787 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.706843 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706804 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.706997 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.707035 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.706997 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.707225 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707207 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.707268 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707226 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.707453 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707433 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.707526 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707454 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.707660 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707645 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.707702 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707660 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.707852 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707833 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.707963 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.707852 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.708055 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708036 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.708095 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708057 2581 scope.go:117] "RemoveContainer" containerID="03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c" Sep 14 04:50:49.708274 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708257 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c"} err="failed to get container status \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": rpc error: code = NotFound desc = could not find container \"03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c\": container with ID starting with 03eed35779801ea576e51fd0412c2e49ff908e7c6706b514149c3a4585aae84c not found: ID does not exist" Sep 14 04:50:49.708344 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708276 2581 scope.go:117] "RemoveContainer" containerID="f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96" Sep 14 04:50:49.708513 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708495 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96"} err="failed to get container status \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": rpc error: code = NotFound desc = could not find container \"f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96\": container with ID starting with f5b1c5861fdbaa1a0b3b28fb5ab3b4686785b6223c3b43e907417dff3572cc96 not found: ID does not exist" Sep 14 04:50:49.708513 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708513 2581 scope.go:117] "RemoveContainer" containerID="cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90" Sep 14 04:50:49.708708 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708690 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90"} err="failed to get container status \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": rpc error: code = NotFound desc = could not find container \"cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90\": container with ID starting with cfb0488ce6ebd8d690c0a83a54640dc25a6bc8b8c9effc7e2b804c1f9664fc90 not found: ID does not exist" Sep 14 04:50:49.708774 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708709 2581 scope.go:117] "RemoveContainer" containerID="95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45" Sep 14 04:50:49.708927 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708905 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45"} err="failed to get container status \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": rpc error: code = NotFound desc = could not find container \"95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45\": container with ID starting with 95e2eb1ff14c622e50507eea2b476820cbba6aeb1309c0e777c9df1515ba9f45 not found: ID does not exist" Sep 14 04:50:49.708967 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.708928 2581 scope.go:117] "RemoveContainer" containerID="762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882" Sep 14 04:50:49.709151 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.709133 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882"} err="failed to get container status \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": rpc error: code = NotFound desc = could not find container \"762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882\": container with ID starting with 762fd482aebd5d7c0032158ccd50ed90f8d9446e1dc286bce7cce987a4599882 not found: ID does not exist" Sep 14 04:50:49.709214 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.709153 2581 scope.go:117] "RemoveContainer" containerID="325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381" Sep 14 04:50:49.709367 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.709350 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381"} err="failed to get container status \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": rpc error: code = NotFound desc = could not find container \"325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381\": container with ID starting with 325ffdd3fc576d743d976aeaf5ac944f88552833f1ee6e057cd917a146c37381 not found: ID does not exist" Sep 14 04:50:49.709409 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.709367 2581 scope.go:117] "RemoveContainer" containerID="21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd" Sep 14 04:50:49.709569 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.709552 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd"} err="failed to get container status \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": rpc error: code = NotFound desc = could not find container \"21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd\": container with ID starting with 21d552aefc527e0a628132de1e60e033a140f2acbe4cc5fb8d87ba83ebcae0bd not found: ID does not exist" Sep 14 04:50:49.721253 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721227 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:49.721518 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721507 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="prometheus" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721520 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="prometheus" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721530 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="config-reloader" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721536 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="config-reloader" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721542 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-thanos" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721548 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-thanos" Sep 14 04:50:49.721563 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721561 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="init-config-reloader" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721566 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="init-config-reloader" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721572 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="thanos-sidecar" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721578 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="thanos-sidecar" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721587 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-web" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721593 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-web" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721604 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721610 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721654 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="prometheus" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721660 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-thanos" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721668 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy-web" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721676 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="config-reloader" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721682 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="thanos-sidecar" Sep 14 04:50:49.721731 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.721687 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" containerName="kube-rbac-proxy" Sep 14 04:50:49.726661 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.726644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.729048 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729024 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 04:50:49.729230 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 04:50:49.729230 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729198 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 04:50:49.729514 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729498 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 04:50:49.729590 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729556 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 04:50:49.729643 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729584 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 04:50:49.729844 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729817 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-69dn3k14cp6iv\"" Sep 14 04:50:49.729910 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729894 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 04:50:49.729910 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.729903 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 04:50:49.730254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.730238 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 04:50:49.730323 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.730271 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mldrm\"" Sep 14 04:50:49.730489 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.730472 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 04:50:49.730544 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.730472 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 04:50:49.732551 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.732520 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 04:50:49.734921 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.734904 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 04:50:49.737234 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.737218 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 04:50:49.745701 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.745682 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:49.874231 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874231 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-web-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874277 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nvq\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-kube-api-access-h6nvq\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874590 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874656 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config-out\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874705 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.874931 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.874786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975322 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975285 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975322 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config-out\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975828 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.975828 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.975618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976224 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976188 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976224 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976255 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-web-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976392 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nvq\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-kube-api-access-h6nvq\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.976610 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.976581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.977872 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.977540 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.978469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.978419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.979108 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.979082 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.979276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.979230 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-web-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.979633 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.979608 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config-out\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980154 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980270 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980175 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980463 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980801 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980779 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-config\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980910 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980776 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980972 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.980972 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.980403 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.981340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.981313 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.981340 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.981331 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.981450 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.981376 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.981681 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.981664 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.981972 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.981955 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:49.991245 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:49.991224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nvq\" (UniqueName: \"kubernetes.io/projected/45d054fb-2a11-4d82-b0ad-87ee0ae0ba50-kube-api-access-h6nvq\") pod \"prometheus-k8s-0\" (UID: \"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:50.036607 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:50.036569 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:50:50.326865 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:50.326792 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 04:50:50.330129 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:50:50.330084 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d054fb_2a11_4d82_b0ad_87ee0ae0ba50.slice/crio-ff87fb92aa0082ffa94ca935d4bc9662856017cf4828dc790a537ebe2bf0a9b2 WatchSource:0}: Error finding container ff87fb92aa0082ffa94ca935d4bc9662856017cf4828dc790a537ebe2bf0a9b2: Status 404 returned error can't find the container with id ff87fb92aa0082ffa94ca935d4bc9662856017cf4828dc790a537ebe2bf0a9b2 Sep 14 04:50:50.652786 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:50.652753 2581 generic.go:356] "Generic (PLEG): container finished" podID="45d054fb-2a11-4d82-b0ad-87ee0ae0ba50" containerID="0d12a8e580523b2728efc419ce19bafaf02dcc96f30df0859c143c7e889557c0" exitCode=0 Sep 14 04:50:50.653217 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:50.652834 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerDied","Data":"0d12a8e580523b2728efc419ce19bafaf02dcc96f30df0859c143c7e889557c0"} Sep 14 04:50:50.653217 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:50.652932 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"ff87fb92aa0082ffa94ca935d4bc9662856017cf4828dc790a537ebe2bf0a9b2"} Sep 14 04:50:51.552595 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.552563 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fe62de-8648-42d9-880f-fef43aadd2a5" path="/var/lib/kubelet/pods/f2fe62de-8648-42d9-880f-fef43aadd2a5/volumes" Sep 14 04:50:51.661838 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661803 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"7c77a048480fc7c022a2d59d421e5e37d0feca8cf73e39fde127e1948ac80c72"} Sep 14 04:50:51.662277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"6d26b8d624eafff2ea203bdf2b52cbf63b4eda9145a2be7e49744fc701d20c66"} Sep 14 04:50:51.662277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661888 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"75d230e52d9e7014b08846d0c599c69a848168ada4b7ff0cfe079e2e134c84cd"} Sep 14 04:50:51.662277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"982970798cc30f9163fea60f98e8b9a5dc357c9dd453d2462dd5de5134db1084"} Sep 14 04:50:51.662277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"2e8a2f3585f7a003caafb3e9376901a594f3e44c837696b2c4ffeda27275f9eb"} Sep 14 04:50:51.662277 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.661957 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45d054fb-2a11-4d82-b0ad-87ee0ae0ba50","Type":"ContainerStarted","Data":"c9df4916ee70db7dfc0c42320f7f7ec86cb8a5478f682588ca0031444c1c218f"} Sep 14 04:50:51.697463 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:51.697411 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.674443596 podStartE2EDuration="2.697395473s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.951877ms" isStatefulPod=false podCreationTimestamp="2026-09-14 04:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:50:51.695519999 +0000 UTC m=+262.653136750" watchObservedRunningTime="2026-09-14 04:50:51.697395473 +0000 UTC m=+262.655012253" Sep 14 04:50:55.036964 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:50:55.036891 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:51:29.472506 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:51:29.472484 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 04:51:50.037014 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:51:50.036970 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:51:50.052282 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:51:50.052259 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:51:50.901629 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:51:50.901604 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 04:52:44.638216 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.638140 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk"] Sep 14 04:52:44.641415 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.641399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.644088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.644063 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 04:52:44.644088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.644078 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 14 04:52:44.644088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.644068 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 04:52:44.644319 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.644081 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 04:52:44.652628 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.652608 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk"] Sep 14 04:52:44.730758 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.730734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98b3799a-05c5-44c1-99a3-20a18e7ccf89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.730864 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.730769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/98b3799a-05c5-44c1-99a3-20a18e7ccf89-kube-api-access-zntnn\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.832050 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.832016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/98b3799a-05c5-44c1-99a3-20a18e7ccf89-kube-api-access-zntnn\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.832187 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.832084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98b3799a-05c5-44c1-99a3-20a18e7ccf89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.834525 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.834497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98b3799a-05c5-44c1-99a3-20a18e7ccf89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.840601 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.840579 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/98b3799a-05c5-44c1-99a3-20a18e7ccf89-kube-api-access-zntnn\") pod \"managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk\" (UID: \"98b3799a-05c5-44c1-99a3-20a18e7ccf89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:44.960793 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:44.960740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" Sep 14 04:52:45.233145 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:45.232964 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk"] Sep 14 04:52:45.235955 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:52:45.235927 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98b3799a_05c5_44c1_99a3_20a18e7ccf89.slice/crio-a6b0dfc7c7b0d386fc64929eef7f64332eda0a6ea45a87e756ded9d00c862170 WatchSource:0}: Error finding container a6b0dfc7c7b0d386fc64929eef7f64332eda0a6ea45a87e756ded9d00c862170: Status 404 returned error can't find the container with id a6b0dfc7c7b0d386fc64929eef7f64332eda0a6ea45a87e756ded9d00c862170 Sep 14 04:52:45.237959 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:45.237944 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 04:52:46.093905 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:46.093869 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" event={"ID":"98b3799a-05c5-44c1-99a3-20a18e7ccf89","Type":"ContainerStarted","Data":"a6b0dfc7c7b0d386fc64929eef7f64332eda0a6ea45a87e756ded9d00c862170"} Sep 14 04:52:49.107822 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:49.107782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" event={"ID":"98b3799a-05c5-44c1-99a3-20a18e7ccf89","Type":"ContainerStarted","Data":"753719e7a718bb9ecc3fc6676d305f23f28f6dfd57f7fdb56ffd5bed2b4c89a0"} Sep 14 04:52:49.126467 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:52:49.126413 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4d54f5c4-zrrgk" podStartSLOduration=2.093127277 podStartE2EDuration="5.126395514s" totalImagesPullingTime="3.033268237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:52:49.12576569 +0000 UTC m=+380.083382430" watchObservedRunningTime="2026-09-14 04:52:49.126395514 +0000 UTC m=+380.084012291" Sep 14 04:54:01.596023 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.595943 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nf4wc"] Sep 14 04:54:01.599149 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.599111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.600861 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.600841 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ggdgg\"" Sep 14 04:54:01.600978 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.600842 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 14 04:54:01.601166 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.601147 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 14 04:54:01.606961 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.606934 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nf4wc"] Sep 14 04:54:01.675068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.675032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrlc\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-kube-api-access-2xrlc\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.675068 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.675069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.775884 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.775843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrlc\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-kube-api-access-2xrlc\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.776099 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.775894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.784934 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.784908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.785099 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.785080 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrlc\" (UniqueName: \"kubernetes.io/projected/02dc1537-23ed-4284-b5f8-fd0f7a9cfc29-kube-api-access-2xrlc\") pod \"cert-manager-webhook-6486ffd997-nf4wc\" (UID: \"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:01.908409 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:01.908372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:02.178388 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:02.178311 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nf4wc"] Sep 14 04:54:02.181525 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:54:02.181501 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02dc1537_23ed_4284_b5f8_fd0f7a9cfc29.slice/crio-fe675ced80f1a75f2815a28d9af72309a0bb340475a65ba860f71aa344321f45 WatchSource:0}: Error finding container fe675ced80f1a75f2815a28d9af72309a0bb340475a65ba860f71aa344321f45: Status 404 returned error can't find the container with id fe675ced80f1a75f2815a28d9af72309a0bb340475a65ba860f71aa344321f45 Sep 14 04:54:02.398407 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:02.398370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" event={"ID":"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29","Type":"ContainerStarted","Data":"fe675ced80f1a75f2815a28d9af72309a0bb340475a65ba860f71aa344321f45"} Sep 14 04:54:05.414319 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:05.414285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" event={"ID":"02dc1537-23ed-4284-b5f8-fd0f7a9cfc29","Type":"ContainerStarted","Data":"8a225e7a441e2e8cd087fb273ec265144e69c6cec476697fe938a5e64b9e3acd"} Sep 14 04:54:05.414696 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:05.414328 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:11.421845 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:11.421809 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" Sep 14 04:54:11.439264 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:11.439216 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nf4wc" podStartSLOduration=7.606293525 podStartE2EDuration="10.439199078s" totalImagesPullingTime="2.832905553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:05.438934269 +0000 UTC m=+456.396551008" watchObservedRunningTime="2026-09-14 04:54:11.439199078 +0000 UTC m=+462.396815815" Sep 14 04:54:34.063190 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.063158 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497"] Sep 14 04:54:34.066669 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.066650 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.068288 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.068262 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 04:54:34.068288 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.068281 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 04:54:34.068575 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.068559 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q4vrb\"" Sep 14 04:54:34.068823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.068804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 04:54:34.068904 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.068867 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 04:54:34.083764 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.083742 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497"] Sep 14 04:54:34.139432 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.139408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.139525 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.139438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.139525 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.139474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.139525 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.139499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.139627 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.139546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xl2w\" (UniqueName: \"kubernetes.io/projected/fe6c4506-3bff-4596-a594-d29c984f92d5-kube-api-access-8xl2w\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.240716 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.240695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.240783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.240724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.240783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.240752 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.240783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.240767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.240937 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.240795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xl2w\" (UniqueName: \"kubernetes.io/projected/fe6c4506-3bff-4596-a594-d29c984f92d5-kube-api-access-8xl2w\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.241179 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.241157 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.243013 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.242987 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe6c4506-3bff-4596-a594-d29c984f92d5-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.243163 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.243146 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.243232 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.243217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe6c4506-3bff-4596-a594-d29c984f92d5-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.259456 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.259431 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xl2w\" (UniqueName: \"kubernetes.io/projected/fe6c4506-3bff-4596-a594-d29c984f92d5-kube-api-access-8xl2w\") pod \"opendatahub-operator-controller-manager-78b6456dc5-xh497\" (UID: \"fe6c4506-3bff-4596-a594-d29c984f92d5\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.377175 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.377153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:34.666867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:34.666796 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497"] Sep 14 04:54:34.669721 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:54:34.669696 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6c4506_3bff_4596_a594_d29c984f92d5.slice/crio-71e9a0c7702a7f1469281b916d3dafae1e21a2c3750ce6bbb964f94a1b8bb805 WatchSource:0}: Error finding container 71e9a0c7702a7f1469281b916d3dafae1e21a2c3750ce6bbb964f94a1b8bb805: Status 404 returned error can't find the container with id 71e9a0c7702a7f1469281b916d3dafae1e21a2c3750ce6bbb964f94a1b8bb805 Sep 14 04:54:35.534132 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:35.534079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" event={"ID":"fe6c4506-3bff-4596-a594-d29c984f92d5","Type":"ContainerStarted","Data":"71e9a0c7702a7f1469281b916d3dafae1e21a2c3750ce6bbb964f94a1b8bb805"} Sep 14 04:54:38.548689 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:38.548650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" event={"ID":"fe6c4506-3bff-4596-a594-d29c984f92d5","Type":"ContainerStarted","Data":"be98355552a00df254de3ed15725eb8201c79e8aab4fc69f12292fdbe5f49242"} Sep 14 04:54:39.554775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:39.554735 2581 generic.go:356] "Generic (PLEG): container finished" podID="fe6c4506-3bff-4596-a594-d29c984f92d5" containerID="be98355552a00df254de3ed15725eb8201c79e8aab4fc69f12292fdbe5f49242" exitCode=0 Sep 14 04:54:39.555239 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:39.554817 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" event={"ID":"fe6c4506-3bff-4596-a594-d29c984f92d5","Type":"ContainerDied","Data":"be98355552a00df254de3ed15725eb8201c79e8aab4fc69f12292fdbe5f49242"} Sep 14 04:54:40.561361 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:40.561329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" event={"ID":"fe6c4506-3bff-4596-a594-d29c984f92d5","Type":"ContainerStarted","Data":"3a521094ad3cb01ff99965f18e632cdeaf4b266685134db77eed976683a7b121"} Sep 14 04:54:40.561751 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:40.561436 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:40.606060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:40.605990 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" podStartSLOduration=2.362295636 podStartE2EDuration="6.605970946s" totalImagesPullingTime="3.19539436s" totalInitContainerRuntime="1.04828095s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:40.605808711 +0000 UTC m=+491.563425449" watchObservedRunningTime="2026-09-14 04:54:40.605970946 +0000 UTC m=+491.563587683" Sep 14 04:54:51.568254 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:51.568220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-xh497" Sep 14 04:54:55.702662 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.702630 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-jnksg"] Sep 14 04:54:55.705894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.705877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.712996 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.712970 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 04:54:55.713286 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.713266 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 04:54:55.713413 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.713300 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 04:54:55.713413 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.713313 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 04:54:55.713504 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.713424 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tmngr\"" Sep 14 04:54:55.713973 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.713957 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 04:54:55.737313 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.737289 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-jnksg"] Sep 14 04:54:55.813271 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.813245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7b6b\" (UniqueName: \"kubernetes.io/projected/98ec6ce5-0cff-4b36-a85d-735c65f359df-kube-api-access-p7b6b\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.813385 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.813299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.813385 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.813322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-metrics-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.813385 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.813345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98ec6ce5-0cff-4b36-a85d-735c65f359df-manager-config\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.914600 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.914572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7b6b\" (UniqueName: \"kubernetes.io/projected/98ec6ce5-0cff-4b36-a85d-735c65f359df-kube-api-access-p7b6b\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.914703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.914626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.914703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.914652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-metrics-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.914703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.914678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98ec6ce5-0cff-4b36-a85d-735c65f359df-manager-config\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.915444 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.915422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98ec6ce5-0cff-4b36-a85d-735c65f359df-manager-config\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.917149 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.917111 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.917418 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.917400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ec6ce5-0cff-4b36-a85d-735c65f359df-metrics-cert\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:55.925082 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:55.925058 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7b6b\" (UniqueName: \"kubernetes.io/projected/98ec6ce5-0cff-4b36-a85d-735c65f359df-kube-api-access-p7b6b\") pod \"lws-controller-manager-5854c46596-jnksg\" (UID: \"98ec6ce5-0cff-4b36-a85d-735c65f359df\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:56.020338 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:56.020278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:56.351294 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:54:56.351264 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec6ce5_0cff_4b36_a85d_735c65f359df.slice/crio-eb030420cf89474787efa46fcc6c389ff416bf5579f80748a6ee0bec2e8e8dfa WatchSource:0}: Error finding container eb030420cf89474787efa46fcc6c389ff416bf5579f80748a6ee0bec2e8e8dfa: Status 404 returned error can't find the container with id eb030420cf89474787efa46fcc6c389ff416bf5579f80748a6ee0bec2e8e8dfa Sep 14 04:54:56.353760 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:56.353633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-jnksg"] Sep 14 04:54:56.630649 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:56.630621 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" event={"ID":"98ec6ce5-0cff-4b36-a85d-735c65f359df","Type":"ContainerStarted","Data":"eb030420cf89474787efa46fcc6c389ff416bf5579f80748a6ee0bec2e8e8dfa"} Sep 14 04:54:58.641850 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:58.641821 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" event={"ID":"98ec6ce5-0cff-4b36-a85d-735c65f359df","Type":"ContainerStarted","Data":"064135b0499278aad5f3e96d7e7c7b3cfce0c6b7ead852e0c4a806b848fb347a"} Sep 14 04:54:58.642226 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:58.641927 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:54:58.663327 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:54:58.663287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" podStartSLOduration=1.442016311 podStartE2EDuration="3.663272255s" totalImagesPullingTime="2.221255944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:58.661347174 +0000 UTC m=+509.618963911" watchObservedRunningTime="2026-09-14 04:54:58.663272255 +0000 UTC m=+509.620888992" Sep 14 04:55:02.077273 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.077245 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:02.080675 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.080656 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.082925 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.082902 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 04:55:02.083210 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.083186 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 04:55:02.083303 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.083267 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bksdw\"" Sep 14 04:55:02.083354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.083267 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 04:55:02.093374 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.093354 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:02.169292 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169404 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169404 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169404 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169529 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169529 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k882c\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169529 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169643 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.169643 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.169584 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270218 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270334 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270334 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270334 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270267 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k882c\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270334 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270543 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270543 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270543 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270543 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270758 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270600 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270758 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270758 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.270854 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.270787 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.271073 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.271049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.272590 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.272569 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.272671 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.272578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.280030 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.280008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.280109 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.280071 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k882c\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckvq9f\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.344436 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.344378 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d"] Sep 14 04:55:02.348559 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.348542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.366330 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.366310 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d"] Sep 14 04:55:02.390316 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.390288 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:02.472396 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472538 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472538 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472638 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472638 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472638 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472638 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472782 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd6b\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-kube-api-access-5vd6b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.472782 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.472725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573742 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd6b\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-kube-api-access-5vd6b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.573880 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574217 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574217 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.573977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574315 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.574225 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574355 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.574328 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574407 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.574385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574442 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.574400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.574644 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.574627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.576161 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.576142 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.576434 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.576418 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.582216 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.582198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd6b\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-kube-api-access-5vd6b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.582477 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.582459 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d\" (UID: \"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.658359 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.658336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:02.665824 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.665802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:02.668850 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:55:02.668826 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38cdbc4_5ca5_4e70_ad4e_06fe3da05d75.slice/crio-f98334d2f4c05ddd66a8bad1e1f9fe15c73450cb269bd57cac4ee0348076a1d4 WatchSource:0}: Error finding container f98334d2f4c05ddd66a8bad1e1f9fe15c73450cb269bd57cac4ee0348076a1d4: Status 404 returned error can't find the container with id f98334d2f4c05ddd66a8bad1e1f9fe15c73450cb269bd57cac4ee0348076a1d4 Sep 14 04:55:02.936317 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:02.936265 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d"] Sep 14 04:55:02.938405 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:55:02.938378 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6ae42e_2592_423c_8d2c_a7bfdfb9bb23.slice/crio-295b900bfaaa5cbe047c30122f83242b5684a115a1f5f3d3528c3be6b417833d WatchSource:0}: Error finding container 295b900bfaaa5cbe047c30122f83242b5684a115a1f5f3d3528c3be6b417833d: Status 404 returned error can't find the container with id 295b900bfaaa5cbe047c30122f83242b5684a115a1f5f3d3528c3be6b417833d Sep 14 04:55:03.666173 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:03.666111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" event={"ID":"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75","Type":"ContainerStarted","Data":"f98334d2f4c05ddd66a8bad1e1f9fe15c73450cb269bd57cac4ee0348076a1d4"} Sep 14 04:55:03.667423 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:03.667385 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" event={"ID":"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23","Type":"ContainerStarted","Data":"295b900bfaaa5cbe047c30122f83242b5684a115a1f5f3d3528c3be6b417833d"} Sep 14 04:55:04.514556 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.514516 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-q4kkd"] Sep 14 04:55:04.519728 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.519697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.523745 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.522234 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vr2n2\"" Sep 14 04:55:04.523745 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.522509 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 14 04:55:04.530787 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.529962 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-q4kkd"] Sep 14 04:55:04.694500 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.694430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-cert\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.694957 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.694722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75598\" (UniqueName: \"kubernetes.io/projected/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-kube-api-access-75598\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.795681 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.795606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75598\" (UniqueName: \"kubernetes.io/projected/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-kube-api-access-75598\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.795681 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.795652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-cert\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.798483 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.798445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-cert\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.806241 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.806214 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75598\" (UniqueName: \"kubernetes.io/projected/0bec66f9-3d2e-4dd7-bb7b-daa71e966e45-kube-api-access-75598\") pod \"odh-model-controller-7998c67dc4-q4kkd\" (UID: \"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45\") " pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:04.835872 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:04.835842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:05.834467 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.834425 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:05.834894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.834512 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:05.834894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.834545 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:05.840524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.840495 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:05.840627 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.840568 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:05.840627 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:05.840601 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:06.017521 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.017481 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-q4kkd"] Sep 14 04:55:06.020665 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:55:06.020635 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bec66f9_3d2e_4dd7_bb7b_daa71e966e45.slice/crio-1fa799d66a6ca77408c76a4d73deb361275e1a8d65827082df2add80f26bde6f WatchSource:0}: Error finding container 1fa799d66a6ca77408c76a4d73deb361275e1a8d65827082df2add80f26bde6f: Status 404 returned error can't find the container with id 1fa799d66a6ca77408c76a4d73deb361275e1a8d65827082df2add80f26bde6f Sep 14 04:55:06.685580 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.685545 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" event={"ID":"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45","Type":"ContainerStarted","Data":"1fa799d66a6ca77408c76a4d73deb361275e1a8d65827082df2add80f26bde6f"} Sep 14 04:55:06.687546 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.687516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" event={"ID":"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75","Type":"ContainerStarted","Data":"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89"} Sep 14 04:55:06.689310 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.689288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" event={"ID":"0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23","Type":"ContainerStarted","Data":"39e80dab6ff9c6a1b5bd53fbd111e65d461ea104a379b17d06ecc8147b3fc73c"} Sep 14 04:55:06.711131 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.711074 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" podStartSLOduration=1.547654383 podStartE2EDuration="4.711055603s" totalImagesPullingTime="3.16340122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:55:06.709039175 +0000 UTC m=+517.666655913" watchObservedRunningTime="2026-09-14 04:55:06.711055603 +0000 UTC m=+517.668672341" Sep 14 04:55:06.731208 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:06.731168 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" podStartSLOduration=1.8312626029999999 podStartE2EDuration="4.731153445s" totalImagesPullingTime="2.899890842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:55:06.730516232 +0000 UTC m=+517.688132970" watchObservedRunningTime="2026-09-14 04:55:06.731153445 +0000 UTC m=+517.688770191" Sep 14 04:55:07.391067 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.391028 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:07.392596 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.392572 2581 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 14 04:55:07.392714 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.392627 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 14 04:55:07.659409 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.659338 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:07.664501 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.664477 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:07.694733 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.694697 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:07.695648 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.695613 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d" Sep 14 04:55:07.749932 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:07.749903 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:08.391276 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:08.391243 2581 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 14 04:55:08.391690 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:08.391306 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 14 04:55:09.391424 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.391388 2581 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 14 04:55:09.391785 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.391451 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 14 04:55:09.648459 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.648397 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5854c46596-jnksg" Sep 14 04:55:09.703322 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.703287 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" event={"ID":"0bec66f9-3d2e-4dd7-bb7b-daa71e966e45","Type":"ContainerStarted","Data":"c2c02c22f120cf0063b6fb792a97d2687f47e3f74471922957a9e5ca0829fc4c"} Sep 14 04:55:09.703540 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.703420 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:55:09.703540 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.703499 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" containerID="cri-o://8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89" gracePeriod=30 Sep 14 04:55:09.737553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:09.737506 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" podStartSLOduration=3.114131124 podStartE2EDuration="5.737492019s" totalImagesPullingTime="2.623360895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:55:09.736485904 +0000 UTC m=+520.694102641" watchObservedRunningTime="2026-09-14 04:55:09.737492019 +0000 UTC m=+520.695108775" Sep 14 04:55:11.845514 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.845479 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w"] Sep 14 04:55:11.855573 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.855547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.857835 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.857810 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 14 04:55:11.857957 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.857821 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v8cmr\"" Sep 14 04:55:11.858420 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.858398 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 14 04:55:11.859088 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.859068 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 14 04:55:11.867184 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.867110 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w"] Sep 14 04:55:11.960810 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54dd\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-kube-api-access-h54dd\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.960810 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.961041 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.961041 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960863 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.961041 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.961041 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.960995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/259a958e-0ad5-42c1-9395-051d7da3ab9d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:11.961041 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:11.961030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.061768 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.061737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.061923 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.061791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/259a958e-0ad5-42c1-9395-051d7da3ab9d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.061923 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.061912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.062031 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.061972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54dd\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-kube-api-access-h54dd\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.062031 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.062003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.062031 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.062022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.062228 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.062050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.062823 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.062791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.064588 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.064390 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/259a958e-0ad5-42c1-9395-051d7da3ab9d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.064719 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.064654 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.064719 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.064708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.064902 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.064882 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.074296 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.074271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54dd\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-kube-api-access-h54dd\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.074733 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.074713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259a958e-0ad5-42c1-9395-051d7da3ab9d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h564w\" (UID: \"259a958e-0ad5-42c1-9395-051d7da3ab9d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.166656 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.166633 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:12.442072 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.442000 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w"] Sep 14 04:55:12.444919 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:55:12.444889 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259a958e_0ad5_42c1_9395_051d7da3ab9d.slice/crio-d65d1bc889b5005a28426cd28b3e760fdd01e0d3ff688fa0b50da2674d4004c6 WatchSource:0}: Error finding container d65d1bc889b5005a28426cd28b3e760fdd01e0d3ff688fa0b50da2674d4004c6: Status 404 returned error can't find the container with id d65d1bc889b5005a28426cd28b3e760fdd01e0d3ff688fa0b50da2674d4004c6 Sep 14 04:55:12.718171 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:12.718068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" event={"ID":"259a958e-0ad5-42c1-9395-051d7da3ab9d","Type":"ContainerStarted","Data":"d65d1bc889b5005a28426cd28b3e760fdd01e0d3ff688fa0b50da2674d4004c6"} Sep 14 04:55:14.941800 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:14.941752 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:14.942206 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:14.941844 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:55:15.037150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.037108 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:15.095465 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095428 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095483 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k882c\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095540 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095596 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095652 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095624 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095678 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095700 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:55:15.095867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.095867 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095831 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert\") pod \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\" (UID: \"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75\") " Sep 14 04:55:15.096150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.095939 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:55:15.096150 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096066 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:55:15.096265 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096186 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.096265 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096206 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.096265 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096222 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.096425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096318 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data" (OuterVolumeSpecName: "istio-data") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:55:15.096425 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.096356 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 04:55:15.098363 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.098335 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c" (OuterVolumeSpecName: "kube-api-access-k882c") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "kube-api-access-k882c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:55:15.098490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.098386 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 04:55:15.098490 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.098417 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token" (OuterVolumeSpecName: "istio-token") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 04:55:15.098813 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.098789 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" (UID: "a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 04:55:15.196887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196858 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.196887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196883 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.196887 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196892 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.197085 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196901 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.197085 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196910 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k882c\" (UniqueName: \"kubernetes.io/projected/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-kube-api-access-k882c\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.197085 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.196920 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 04:55:15.736947 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.736913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" event={"ID":"259a958e-0ad5-42c1-9395-051d7da3ab9d","Type":"ContainerStarted","Data":"0416d615a2979a95e356ee5a11f0ebeaa31620e4207f90416ec05de65084591e"} Sep 14 04:55:15.737148 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.737022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:15.738322 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.738298 2581 generic.go:356] "Generic (PLEG): container finished" podID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerID="8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89" exitCode=0 Sep 14 04:55:15.738426 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.738337 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" Sep 14 04:55:15.738426 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.738345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" event={"ID":"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75","Type":"ContainerDied","Data":"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89"} Sep 14 04:55:15.738426 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.738376 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f" event={"ID":"a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75","Type":"ContainerDied","Data":"f98334d2f4c05ddd66a8bad1e1f9fe15c73450cb269bd57cac4ee0348076a1d4"} Sep 14 04:55:15.738426 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.738399 2581 scope.go:117] "RemoveContainer" containerID="8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89" Sep 14 04:55:15.746725 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.746709 2581 scope.go:117] "RemoveContainer" containerID="8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89" Sep 14 04:55:15.746995 ip-10-0-132-238 kubenswrapper[2581]: E0914 04:55:15.746978 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89\": container with ID starting with 8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89 not found: ID does not exist" containerID="8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89" Sep 14 04:55:15.747040 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.747003 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89"} err="failed to get container status \"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89\": rpc error: code = NotFound desc = could not find container \"8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89\": container with ID starting with 8481dd2943d5d147338ee4a4cd345f516947f63bd2fabaeac580a0a0c429aa89 not found: ID does not exist" Sep 14 04:55:15.764763 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.764722 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" podStartSLOduration=2.273130806 podStartE2EDuration="4.764709314s" totalImagesPullingTime="2.491578508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:55:15.762820441 +0000 UTC m=+526.720437178" watchObservedRunningTime="2026-09-14 04:55:15.764709314 +0000 UTC m=+526.722326043" Sep 14 04:55:15.790242 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.790218 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:15.800909 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:15.800891 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckvq9f"] Sep 14 04:55:16.747108 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:16.747077 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h564w" Sep 14 04:55:17.553336 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:17.553304 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" path="/var/lib/kubelet/pods/a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75/volumes" Sep 14 04:55:20.709946 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:55:20.709919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-q4kkd" Sep 14 04:57:14.196359 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.196316 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4"] Sep 14 04:57:14.197051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.196903 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" Sep 14 04:57:14.197051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.196924 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" Sep 14 04:57:14.197051 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.197042 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38cdbc4-5ca5-4e70-ad4e-06fe3da05d75" containerName="istio-proxy" Sep 14 04:57:14.200199 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.200178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.202032 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.201999 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-42f85\"" Sep 14 04:57:14.212046 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.212018 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4"] Sep 14 04:57:14.268025 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.267998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268159 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268159 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268159 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d72ccddb-21db-4e33-bd50-747d4be9d4de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968fl\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-kube-api-access-968fl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.268395 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.268377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369477 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d72ccddb-21db-4e33-bd50-747d4be9d4de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369583 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968fl\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-kube-api-access-968fl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369873 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369873 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.369995 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.369965 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.370212 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.370189 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.370308 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.370218 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.370354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.370313 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.370354 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.370329 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d72ccddb-21db-4e33-bd50-747d4be9d4de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.371832 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.371812 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.372160 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.372140 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.377328 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.377300 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.377439 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.377409 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968fl\" (UniqueName: \"kubernetes.io/projected/d72ccddb-21db-4e33-bd50-747d4be9d4de-kube-api-access-968fl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6jjp4\" (UID: \"d72ccddb-21db-4e33-bd50-747d4be9d4de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.511825 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.511760 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:14.787892 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.787869 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4"] Sep 14 04:57:14.790385 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:57:14.790359 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72ccddb_21db_4e33_bd50_747d4be9d4de.slice/crio-052a7cb82e7dac57bba2d5cd01e60a86be0f4b6bd2fecfbc4b6c69c5951de927 WatchSource:0}: Error finding container 052a7cb82e7dac57bba2d5cd01e60a86be0f4b6bd2fecfbc4b6c69c5951de927: Status 404 returned error can't find the container with id 052a7cb82e7dac57bba2d5cd01e60a86be0f4b6bd2fecfbc4b6c69c5951de927 Sep 14 04:57:14.792435 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.792399 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:57:14.792500 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.792481 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:57:14.792536 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:14.792511 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:57:15.273849 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:15.273812 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" event={"ID":"d72ccddb-21db-4e33-bd50-747d4be9d4de","Type":"ContainerStarted","Data":"5c733a6b231b95c4734362935908cad15cff26cb4696a1542c5a279dd31bcc79"} Sep 14 04:57:15.274292 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:15.273860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" event={"ID":"d72ccddb-21db-4e33-bd50-747d4be9d4de","Type":"ContainerStarted","Data":"052a7cb82e7dac57bba2d5cd01e60a86be0f4b6bd2fecfbc4b6c69c5951de927"} Sep 14 04:57:15.296653 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:15.296606 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podStartSLOduration=1.296591842 podStartE2EDuration="1.296591842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:57:15.294030061 +0000 UTC m=+646.251646797" watchObservedRunningTime="2026-09-14 04:57:15.296591842 +0000 UTC m=+646.254208579" Sep 14 04:57:15.512729 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:15.512700 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:15.517553 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:15.517531 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:16.278443 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:16.278413 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:57:16.279462 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:57:16.279439 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:58:44.084719 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.084640 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5t7fb"] Sep 14 04:58:44.087987 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.087971 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.089817 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.089795 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 04:58:44.089900 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.089815 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-99mff\"" Sep 14 04:58:44.115775 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.115714 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5t7fb"] Sep 14 04:58:44.218350 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.218316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057f9ac5-59ae-410b-8666-a7abd28e0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.218513 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.218378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6h5\" (UniqueName: \"kubernetes.io/projected/057f9ac5-59ae-410b-8666-a7abd28e0120-kube-api-access-2m6h5\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.319134 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.319086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057f9ac5-59ae-410b-8666-a7abd28e0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.319304 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.319177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6h5\" (UniqueName: \"kubernetes.io/projected/057f9ac5-59ae-410b-8666-a7abd28e0120-kube-api-access-2m6h5\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.319848 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.319825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057f9ac5-59ae-410b-8666-a7abd28e0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.327469 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.327445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6h5\" (UniqueName: \"kubernetes.io/projected/057f9ac5-59ae-410b-8666-a7abd28e0120-kube-api-access-2m6h5\") pod \"payload-processing-5b54b7fbb7-5t7fb\" (UID: \"057f9ac5-59ae-410b-8666-a7abd28e0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.398753 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.398713 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:44.694402 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.694335 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5t7fb"] Sep 14 04:58:44.697985 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:58:44.697956 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057f9ac5_59ae_410b_8666_a7abd28e0120.slice/crio-659f61beba23362e8e18a6327199e9efe9a0bc9a81bada000478d21facf97bcf WatchSource:0}: Error finding container 659f61beba23362e8e18a6327199e9efe9a0bc9a81bada000478d21facf97bcf: Status 404 returned error can't find the container with id 659f61beba23362e8e18a6327199e9efe9a0bc9a81bada000478d21facf97bcf Sep 14 04:58:44.699762 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:44.699740 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 04:58:45.699083 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:45.699053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" event={"ID":"057f9ac5-59ae-410b-8666-a7abd28e0120","Type":"ContainerStarted","Data":"659f61beba23362e8e18a6327199e9efe9a0bc9a81bada000478d21facf97bcf"} Sep 14 04:58:46.705784 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:46.705745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" event={"ID":"057f9ac5-59ae-410b-8666-a7abd28e0120","Type":"ContainerStarted","Data":"718f261704d030c118a0a75c77769e23b41421909c4408ce087767046357fc13"} Sep 14 04:58:46.706198 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:46.705805 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:46.724353 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:46.723867 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" podStartSLOduration=1.199428389 podStartE2EDuration="2.723848032s" totalImagesPullingTime="1.524419643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:58:46.723348105 +0000 UTC m=+737.680964844" watchObservedRunningTime="2026-09-14 04:58:46.723848032 +0000 UTC m=+737.681464770" Sep 14 04:58:52.738936 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.738870 2581 generic.go:356] "Generic (PLEG): container finished" podID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerID="5c733a6b231b95c4734362935908cad15cff26cb4696a1542c5a279dd31bcc79" exitCode=0 Sep 14 04:58:52.739263 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.738941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" event={"ID":"d72ccddb-21db-4e33-bd50-747d4be9d4de","Type":"ContainerDied","Data":"5c733a6b231b95c4734362935908cad15cff26cb4696a1542c5a279dd31bcc79"} Sep 14 04:58:52.739263 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.739249 2581 scope.go:117] "RemoveContainer" containerID="5c733a6b231b95c4734362935908cad15cff26cb4696a1542c5a279dd31bcc79" Sep 14 04:58:52.740202 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.740171 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:58:52.740264 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.740249 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:58:52.740302 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:52.740291 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 04:58:53.750324 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:53.749410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" event={"ID":"d72ccddb-21db-4e33-bd50-747d4be9d4de","Type":"ContainerStarted","Data":"c6f2dabbcf16e4d5d77a5c031e24a5facfa6db5b207b9353a60f3a3b0321ffd4"} Sep 14 04:58:54.512837 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:54.512802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:58:54.513044 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:54.513001 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6jjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Sep 14 04:58:54.513147 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:54.513099 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podUID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Sep 14 04:58:55.512213 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:55.512178 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6jjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Sep 14 04:58:55.512616 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:55.512232 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podUID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Sep 14 04:58:56.512678 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:56.512641 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6jjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Sep 14 04:58:56.513073 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:56.512750 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podUID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Sep 14 04:58:57.512856 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:57.512817 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6jjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Sep 14 04:58:57.513241 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:57.512880 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podUID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Sep 14 04:58:57.713969 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:57.713939 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-5t7fb" Sep 14 04:58:58.512471 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:58.512437 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6jjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Sep 14 04:58:58.512642 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:58.512494 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" podUID="d72ccddb-21db-4e33-bd50-747d4be9d4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Sep 14 04:58:59.516078 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:59.516045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:58:59.516466 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:59.516414 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:58:59.516988 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:58:59.516963 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6jjp4" Sep 14 04:59:09.411387 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.411345 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w"] Sep 14 04:59:09.415941 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.415919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.417805 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.417781 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 04:59:09.418209 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.418105 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 14 04:59:09.418209 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.418133 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 04:59:09.418348 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.418280 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c7z9x\"" Sep 14 04:59:09.426525 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.426375 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w"] Sep 14 04:59:09.444415 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.444664 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.444664 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.444664 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444564 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.444664 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfjj\" (UniqueName: \"kubernetes.io/projected/4e33bb41-be6f-451b-81a5-0d3d090c5fca-kube-api-access-cqfjj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.444664 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.444655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545321 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545321 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfjj\" (UniqueName: \"kubernetes.io/projected/4e33bb41-be6f-451b-81a5-0d3d090c5fca-kube-api-access-cqfjj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545524 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545846 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545820 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545936 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.545984 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.545948 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.548014 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.547979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33bb41-be6f-451b-81a5-0d3d090c5fca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.548168 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.548010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33bb41-be6f-451b-81a5-0d3d090c5fca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.553767 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.553748 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfjj\" (UniqueName: \"kubernetes.io/projected/4e33bb41-be6f-451b-81a5-0d3d090c5fca-kube-api-access-cqfjj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ftw7w\" (UID: \"4e33bb41-be6f-451b-81a5-0d3d090c5fca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:09.731456 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:09.731383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:10.007479 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:10.007455 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w"] Sep 14 04:59:10.009599 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:59:10.009569 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e33bb41_be6f_451b_81a5_0d3d090c5fca.slice/crio-e52636b70fc11c3d2607c47033dce35b1e35a5b12f76e126faf3e88ee0ba20df WatchSource:0}: Error finding container e52636b70fc11c3d2607c47033dce35b1e35a5b12f76e126faf3e88ee0ba20df: Status 404 returned error can't find the container with id e52636b70fc11c3d2607c47033dce35b1e35a5b12f76e126faf3e88ee0ba20df Sep 14 04:59:10.837548 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:10.837511 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" event={"ID":"4e33bb41-be6f-451b-81a5-0d3d090c5fca","Type":"ContainerStarted","Data":"e52636b70fc11c3d2607c47033dce35b1e35a5b12f76e126faf3e88ee0ba20df"} Sep 14 04:59:11.843672 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:11.843638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" event={"ID":"4e33bb41-be6f-451b-81a5-0d3d090c5fca","Type":"ContainerStarted","Data":"0ff3fb79f3620f4d7f47076fc0af5862edfd306a95c68e07c47c101138ffbd4d"} Sep 14 04:59:11.864588 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:11.864521 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" podStartSLOduration=1.859288204 podStartE2EDuration="2.864504891s" totalImagesPullingTime="1.005216687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:59:11.86213382 +0000 UTC m=+762.819750548" watchObservedRunningTime="2026-09-14 04:59:11.864504891 +0000 UTC m=+762.822121629" Sep 14 04:59:18.632357 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.632325 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm"] Sep 14 04:59:18.635894 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.635877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.637528 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.637504 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 14 04:59:18.644407 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.644385 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm"] Sep 14 04:59:18.720488 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.720488 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.720703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.720703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqcx\" (UniqueName: \"kubernetes.io/projected/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-kube-api-access-fnqcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.720703 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.720824 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.720708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.821909 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.821875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.821910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.821963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.821985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqcx\" (UniqueName: \"kubernetes.io/projected/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-kube-api-access-fnqcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822060 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.822037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.822062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.822293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.822358 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.822650 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.822404 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.824275 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.824243 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.824522 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.824504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.830011 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.829988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqcx\" (UniqueName: \"kubernetes.io/projected/ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296-kube-api-access-fnqcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm\" (UID: \"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:18.946556 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:18.946504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:19.220290 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.220169 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm"] Sep 14 04:59:19.223207 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:59:19.223177 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbdd262_0ce1_4c26_b4ec_83c3cc0c1296.slice/crio-3374eb0668f7f49eafdd98dae276dfe21df082513f8120b972a706d5bae2d23a WatchSource:0}: Error finding container 3374eb0668f7f49eafdd98dae276dfe21df082513f8120b972a706d5bae2d23a: Status 404 returned error can't find the container with id 3374eb0668f7f49eafdd98dae276dfe21df082513f8120b972a706d5bae2d23a Sep 14 04:59:19.731608 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.731574 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:19.732061 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.731647 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:19.743779 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.743756 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:19.883076 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.883046 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" event={"ID":"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296","Type":"ContainerStarted","Data":"ac4817801ee95539fb488a527144b4468253cba4294a4406fb2341b189a0eb4e"} Sep 14 04:59:19.883238 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.883102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" event={"ID":"ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296","Type":"ContainerStarted","Data":"3374eb0668f7f49eafdd98dae276dfe21df082513f8120b972a706d5bae2d23a"} Sep 14 04:59:19.894352 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.894328 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ftw7w" Sep 14 04:59:19.917462 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:19.917424 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" podStartSLOduration=1.76491095 podStartE2EDuration="1.917411077s" totalImagesPullingTime="152.500127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:59:19.899562136 +0000 UTC m=+770.857178875" watchObservedRunningTime="2026-09-14 04:59:19.917411077 +0000 UTC m=+770.875027814" Sep 14 04:59:28.947629 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:28.947591 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:28.948064 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:28.947655 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:28.960178 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:28.960152 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:29.940203 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:29.940171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm" Sep 14 04:59:54.932210 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:54.932138 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj"] Sep 14 04:59:54.936840 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:54.936818 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:54.938736 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:54.938716 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 14 04:59:54.944781 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:54.944758 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj"] Sep 14 04:59:55.021561 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.021668 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.021668 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514d723b-7e28-4d28-97a4-54dc77771d02-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.021668 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wpl\" (UniqueName: \"kubernetes.io/projected/514d723b-7e28-4d28-97a4-54dc77771d02-kube-api-access-97wpl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.021783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.021783 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.021698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.122938 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.122906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.122938 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.122936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123171 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.122971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514d723b-7e28-4d28-97a4-54dc77771d02-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123171 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.122999 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wpl\" (UniqueName: \"kubernetes.io/projected/514d723b-7e28-4d28-97a4-54dc77771d02-kube-api-access-97wpl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123171 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.123051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123351 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.123184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123351 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.123344 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.123369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.123455 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.123404 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.125229 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.125198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514d723b-7e28-4d28-97a4-54dc77771d02-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.125473 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.125453 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514d723b-7e28-4d28-97a4-54dc77771d02-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.130743 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.130726 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wpl\" (UniqueName: \"kubernetes.io/projected/514d723b-7e28-4d28-97a4-54dc77771d02-kube-api-access-97wpl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cwgkj\" (UID: \"514d723b-7e28-4d28-97a4-54dc77771d02\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.248045 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.247991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 04:59:55.520368 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:55.520344 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj"] Sep 14 04:59:55.522347 ip-10-0-132-238 kubenswrapper[2581]: W0914 04:59:55.522321 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514d723b_7e28_4d28_97a4_54dc77771d02.slice/crio-ba407e547c12ef85fc8c5c1f05b74ccf3821c5d9168b92e15a0d7d13669df4ec WatchSource:0}: Error finding container ba407e547c12ef85fc8c5c1f05b74ccf3821c5d9168b92e15a0d7d13669df4ec: Status 404 returned error can't find the container with id ba407e547c12ef85fc8c5c1f05b74ccf3821c5d9168b92e15a0d7d13669df4ec Sep 14 04:59:56.070801 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:56.070769 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" event={"ID":"514d723b-7e28-4d28-97a4-54dc77771d02","Type":"ContainerStarted","Data":"d03189b485fa555bd66423299ae85e0c9b762a0feafc04a021ee4f94ae007b97"} Sep 14 04:59:56.071169 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:56.070842 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" event={"ID":"514d723b-7e28-4d28-97a4-54dc77771d02","Type":"ContainerStarted","Data":"ba407e547c12ef85fc8c5c1f05b74ccf3821c5d9168b92e15a0d7d13669df4ec"} Sep 14 04:59:56.090014 ip-10-0-132-238 kubenswrapper[2581]: I0914 04:59:56.089964 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" podStartSLOduration=1.9236243119999998 podStartE2EDuration="2.089951531s" totalImagesPullingTime="166.327219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:59:56.088159071 +0000 UTC m=+807.045775805" watchObservedRunningTime="2026-09-14 04:59:56.089951531 +0000 UTC m=+807.047568269" Sep 14 05:00:05.249239 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:00:05.249185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 05:00:05.249658 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:00:05.249254 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 05:00:05.261982 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:00:05.261957 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 05:00:06.130094 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:00:06.130066 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cwgkj" Sep 14 05:01:28.422306 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.422222 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:01:28.426169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.426147 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.427977 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.427957 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-667664-openshift-default-dockercfg-cdhzk\"" Sep 14 05:01:28.437749 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.437722 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:01:28.445001 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.444971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jj9d\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.445371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.445364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.545851 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.545822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.545851 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.545856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.545883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.545929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.545978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546092 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546452 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jj9d\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546452 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546733 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546687 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546850 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.546908 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.546866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.547097 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.547078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.548686 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.548646 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.549156 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.548808 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.560931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.560893 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:01:28.561896 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.561871 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.562002 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.561988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jj9d\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d\") pod \"e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.566009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.565990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.567939 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.567919 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e08402-openshift-default-dockercfg-6f8sx\"" Sep 14 05:01:28.574964 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.574635 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:01:28.647268 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647442 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647442 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647570 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647570 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68lr\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647570 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.647716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.647651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.748753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.748753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.748753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.748962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68lr\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749363 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749363 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749363 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749463 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749378 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.749740 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.749716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.751952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.751929 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.752060 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.752043 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.754809 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.754764 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:28.755371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.755353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68lr\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.755461 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.755444 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token\") pod \"e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:28.886947 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:28.886914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:29.028810 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.028697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:01:29.031747 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:29.031722 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61027dab_03ad_42a6_be36_5ee55a34c36b.slice/crio-cdddc3ef58fec2b655568957280c4112260a6a156364917ac2915811dfb83e73 WatchSource:0}: Error finding container cdddc3ef58fec2b655568957280c4112260a6a156364917ac2915811dfb83e73: Status 404 returned error can't find the container with id cdddc3ef58fec2b655568957280c4112260a6a156364917ac2915811dfb83e73 Sep 14 05:01:29.033824 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.033797 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.033880 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.033858 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.033921 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.033884 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.159837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.159808 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:01:29.161592 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:29.161557 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096381bd_4027_4dbd_93c8_1747155bf0cf.slice/crio-d056e0bdccf27b98d46472ecac8f54601c392fdfa024a33affc94199f5027e7c WatchSource:0}: Error finding container d056e0bdccf27b98d46472ecac8f54601c392fdfa024a33affc94199f5027e7c: Status 404 returned error can't find the container with id d056e0bdccf27b98d46472ecac8f54601c392fdfa024a33affc94199f5027e7c Sep 14 05:01:29.164309 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.164272 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.164410 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.164337 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.164410 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.164369 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:29.531605 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.531562 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerStarted","Data":"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3"} Sep 14 05:01:29.532087 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.531667 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerStarted","Data":"d056e0bdccf27b98d46472ecac8f54601c392fdfa024a33affc94199f5027e7c"} Sep 14 05:01:29.533111 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.533084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerStarted","Data":"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468"} Sep 14 05:01:29.533259 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.533156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerStarted","Data":"cdddc3ef58fec2b655568957280c4112260a6a156364917ac2915811dfb83e73"} Sep 14 05:01:29.552198 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.552094 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podStartSLOduration=1.552078246 podStartE2EDuration="1.552078246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:29.551651087 +0000 UTC m=+900.509267837" watchObservedRunningTime="2026-09-14 05:01:29.552078246 +0000 UTC m=+900.509694984" Sep 14 05:01:29.580402 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.580361 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podStartSLOduration=1.5803483310000002 podStartE2EDuration="1.580348331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:29.578630297 +0000 UTC m=+900.536247039" watchObservedRunningTime="2026-09-14 05:01:29.580348331 +0000 UTC m=+900.537965068" Sep 14 05:01:29.755181 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.755103 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:29.887727 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:29.887698 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:30.760196 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:30.760165 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:30.893457 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:30.893430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:31.544292 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:31.544257 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:31.544498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:31.544323 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:31.545084 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:31.545063 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:31.545314 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:31.545294 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:33.707175 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.707136 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:01:33.710796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.710780 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.712712 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.712687 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-667664\"" Sep 14 05:01:33.712712 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.712701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-667664-dockercfg-mvs58\"" Sep 14 05:01:33.719461 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.719435 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:01:33.801179 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.801135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bgx\" (UniqueName: \"kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.801357 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.801196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.901918 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.901883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bgx\" (UniqueName: \"kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.901918 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.901921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.902648 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.902631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:33.910825 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:33.910791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bgx\" (UniqueName: \"kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx\") pod \"payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:34.022604 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.022517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:34.297781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.297750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:01:34.300918 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:34.300872 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db71e67_e9a1_4665_bb0a_3df67d5f3423.slice/crio-bd04ff6ad513c14ee1f36f569718b6f883819de30dd9e172f46e7f532a9a13fd WatchSource:0}: Error finding container bd04ff6ad513c14ee1f36f569718b6f883819de30dd9e172f46e7f532a9a13fd: Status 404 returned error can't find the container with id bd04ff6ad513c14ee1f36f569718b6f883819de30dd9e172f46e7f532a9a13fd Sep 14 05:01:34.563708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.563595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" event={"ID":"6db71e67-e9a1-4665-bb0a-3df67d5f3423","Type":"ContainerStarted","Data":"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc"} Sep 14 05:01:34.563708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.563664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" event={"ID":"6db71e67-e9a1-4665-bb0a-3df67d5f3423","Type":"ContainerStarted","Data":"bd04ff6ad513c14ee1f36f569718b6f883819de30dd9e172f46e7f532a9a13fd"} Sep 14 05:01:34.563975 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.563804 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:34.579935 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:34.578584 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" podStartSLOduration=1.578566457 podStartE2EDuration="1.578566457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:34.578031921 +0000 UTC m=+905.535648659" watchObservedRunningTime="2026-09-14 05:01:34.578566457 +0000 UTC m=+905.536183195" Sep 14 05:01:35.724288 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.724254 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:01:35.728025 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.728007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.729982 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.729958 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e08402\"" Sep 14 05:01:35.730102 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.729956 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e08402-dockercfg-v85jz\"" Sep 14 05:01:35.739225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.739200 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:01:35.749147 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.749104 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:01:35.753362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.753342 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.761018 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.760994 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:01:35.820642 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.820605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbww\" (UniqueName: \"kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.820804 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.820663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4mz\" (UniqueName: \"kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.820804 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.820765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.820974 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.820862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.924319 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.924284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbww\" (UniqueName: \"kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.924491 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.924326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4mz\" (UniqueName: \"kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.924491 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.924373 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.924491 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.924418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.925078 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.925059 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:35.925141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.925105 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.931692 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.931669 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbww\" (UniqueName: \"kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww\") pod \"payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:35.931788 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:35.931742 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4mz\" (UniqueName: \"kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz\") pod \"payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:36.039791 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.039719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:36.066018 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.065989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:36.315550 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.315509 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:01:36.317671 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:36.317646 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda820576f_886e_4d86_ac7d_88ae43ca9387.slice/crio-7e77f0a4823ec2dd2c390efc591f051479fb07273d81f3005e12b7f18f3cb033 WatchSource:0}: Error finding container 7e77f0a4823ec2dd2c390efc591f051479fb07273d81f3005e12b7f18f3cb033: Status 404 returned error can't find the container with id 7e77f0a4823ec2dd2c390efc591f051479fb07273d81f3005e12b7f18f3cb033 Sep 14 05:01:36.352801 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.352775 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:01:36.355268 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:36.355240 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd771d8_d761_471a_aa98_2bb0110c535a.slice/crio-5ee15603f1606a45950c5315f0f8516a9dbb989cf1039be15f6ad52a96b11c83 WatchSource:0}: Error finding container 5ee15603f1606a45950c5315f0f8516a9dbb989cf1039be15f6ad52a96b11c83: Status 404 returned error can't find the container with id 5ee15603f1606a45950c5315f0f8516a9dbb989cf1039be15f6ad52a96b11c83 Sep 14 05:01:36.575086 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.574981 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" event={"ID":"a820576f-886e-4d86-ac7d-88ae43ca9387","Type":"ContainerStarted","Data":"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2"} Sep 14 05:01:36.575277 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.575214 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:36.575277 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.575255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" event={"ID":"a820576f-886e-4d86-ac7d-88ae43ca9387","Type":"ContainerStarted","Data":"7e77f0a4823ec2dd2c390efc591f051479fb07273d81f3005e12b7f18f3cb033"} Sep 14 05:01:36.576969 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.576933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" event={"ID":"8bd771d8-d761-471a-aa98-2bb0110c535a","Type":"ContainerStarted","Data":"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537"} Sep 14 05:01:36.577144 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.576978 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" event={"ID":"8bd771d8-d761-471a-aa98-2bb0110c535a","Type":"ContainerStarted","Data":"5ee15603f1606a45950c5315f0f8516a9dbb989cf1039be15f6ad52a96b11c83"} Sep 14 05:01:36.577144 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.577038 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:36.595318 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.595271 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" podStartSLOduration=1.595252598 podStartE2EDuration="1.595252598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:36.592660596 +0000 UTC m=+907.550277334" watchObservedRunningTime="2026-09-14 05:01:36.595252598 +0000 UTC m=+907.552869336" Sep 14 05:01:36.609408 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:36.609370 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" podStartSLOduration=1.6093567869999998 podStartE2EDuration="1.609356787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:36.606970646 +0000 UTC m=+907.564587383" watchObservedRunningTime="2026-09-14 05:01:36.609356787 +0000 UTC m=+907.566973524" Sep 14 05:01:38.127799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.127762 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:01:38.132010 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.131986 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.133814 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.133790 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-654e8bde-openshift-default-dockercfg-qvbhk\"" Sep 14 05:01:38.142068 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.142045 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:01:38.249720 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.249914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.249914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.249914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249826 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.249914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.250186 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.249954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.250186 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.250026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbs8\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.250186 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.250064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.250186 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.250091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbs8\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351285 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.351952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.351575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.352633 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.352449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.352774 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.352733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.353421 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.353392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.353697 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.353674 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.354399 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.354375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.356449 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.356426 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.357062 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.357029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.365970 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.365940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbs8\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.366512 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.366491 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token\") pod \"e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.445541 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.445451 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:38.727824 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.727798 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:01:38.730155 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:38.730110 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95706d1e_36cb_4ceb_95d5_a2d34f453439.slice/crio-ed623f670ecc35bf3801271eb7234586652cc733540a9f886381658d1a92a197 WatchSource:0}: Error finding container ed623f670ecc35bf3801271eb7234586652cc733540a9f886381658d1a92a197: Status 404 returned error can't find the container with id ed623f670ecc35bf3801271eb7234586652cc733540a9f886381658d1a92a197 Sep 14 05:01:38.732471 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.732441 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:38.732547 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.732499 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:38.732547 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:38.732526 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:39.593126 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:39.593073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerStarted","Data":"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60"} Sep 14 05:01:39.593508 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:39.593153 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerStarted","Data":"ed623f670ecc35bf3801271eb7234586652cc733540a9f886381658d1a92a197"} Sep 14 05:01:39.612106 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:39.612048 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podStartSLOduration=1.612033087 podStartE2EDuration="1.612033087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:39.610313229 +0000 UTC m=+910.567929966" watchObservedRunningTime="2026-09-14 05:01:39.612033087 +0000 UTC m=+910.569649860" Sep 14 05:01:40.446318 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:40.446279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:40.451311 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:40.451284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:40.600390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:40.600359 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:40.601345 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:40.601325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:01:45.571274 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:45.571237 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:01:46.984850 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:46.984802 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:01:47.001189 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.001163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.003084 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.003023 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:01:47.003587 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.003565 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e7fdd6-openshift-default-dockercfg-6dsz7\"" Sep 14 05:01:47.045694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.045660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.045953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.045933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046105 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046263 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046364 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046515 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdckh\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046761 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.046860 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.046846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.148712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.148783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.148826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.148926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdckh\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.148995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.149024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.149072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.149160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.149233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.149863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.150227 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.150876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.150558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.151248 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.151193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.152723 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.152667 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.156154 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.156134 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.162728 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.162685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdckh\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.167464 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.167420 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.167739 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.167703 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token\") pod \"e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.318431 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.318321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:47.583937 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.583765 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:01:47.584103 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.583949 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:01:47.668414 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.668373 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:01:47.680695 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.680655 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:47.680812 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.680742 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:47.680812 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:47.680782 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:48.693664 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:48.693573 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerStarted","Data":"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b"} Sep 14 05:01:48.694106 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:48.693670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerStarted","Data":"a7f2b2fc8433f0e531e71e7567dbeedb713795e7991e82c2b7ff83084ef3d1a9"} Sep 14 05:01:48.712603 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:48.711992 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podStartSLOduration=2.711973278 podStartE2EDuration="2.711973278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:48.711240752 +0000 UTC m=+919.668857490" watchObservedRunningTime="2026-09-14 05:01:48.711973278 +0000 UTC m=+919.669590015" Sep 14 05:01:49.319349 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:49.319308 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:49.373291 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:49.373243 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:49.700689 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:49.700656 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:49.701736 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:49.701714 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:01:51.713223 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.713189 2581 generic.go:356] "Generic (PLEG): container finished" podID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerID="5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3" exitCode=0 Sep 14 05:01:51.713656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.713265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerDied","Data":"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3"} Sep 14 05:01:51.713781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.713759 2581 scope.go:117] "RemoveContainer" containerID="5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3" Sep 14 05:01:51.715011 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.714979 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:51.715094 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.715054 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:51.715167 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.715096 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:51.715313 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.715292 2581 generic.go:356] "Generic (PLEG): container finished" podID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerID="fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468" exitCode=0 Sep 14 05:01:51.715410 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.715365 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerDied","Data":"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468"} Sep 14 05:01:51.715767 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.715754 2581 scope.go:117] "RemoveContainer" containerID="fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468" Sep 14 05:01:51.716752 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.716729 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:51.716846 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.716793 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:51.716846 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:51.716828 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:01:52.723354 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.723323 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerStarted","Data":"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5"} Sep 14 05:01:52.732780 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.729067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerStarted","Data":"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5"} Sep 14 05:01:52.761141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.756286 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:52.761141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.757210 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:52.761141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.757268 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:52.887321 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.887270 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:52.887769 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.887735 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:52.887889 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:52.887795 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:53.755468 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:53.755429 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:53.756086 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:53.756048 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:53.887594 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:53.887548 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:53.887797 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:53.887621 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:54.755249 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:54.755214 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:54.755575 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:54.755543 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:54.888715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:54.888675 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:54.889032 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:54.888989 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:55.671167 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.662411 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:01:55.694142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.690658 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:01:55.694142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.690778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.699031 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.698503 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e7fdd6\"" Sep 14 05:01:55.699031 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.698848 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e7fdd6-dockercfg-chprd\"" Sep 14 05:01:55.759258 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.759220 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:55.759843 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.759808 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:55.790896 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.790857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.791378 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.791357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqrl\" (UniqueName: \"kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.887866 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.887833 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:55.888895 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.888842 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:55.893062 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.893019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqrl\" (UniqueName: \"kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.893321 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.893303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.894620 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.894589 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:55.904934 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:55.904884 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqrl\" (UniqueName: \"kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl\") pod \"payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:56.017396 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.017305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:56.587193 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.580953 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:01:56.601472 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:01:56.601404 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41de4b1f_5003_4dd7_a72f_e52df4c2ef43.slice/crio-0b664c648e80562d30c1e4d019ca2a6160e6c5424e7f9e6a3d3d3253ac03f835 WatchSource:0}: Error finding container 0b664c648e80562d30c1e4d019ca2a6160e6c5424e7f9e6a3d3d3253ac03f835: Status 404 returned error can't find the container with id 0b664c648e80562d30c1e4d019ca2a6160e6c5424e7f9e6a3d3d3253ac03f835 Sep 14 05:01:56.756197 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.755709 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:56.756197 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.755777 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:56.779058 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.778989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" event={"ID":"41de4b1f-5003-4dd7-a72f-e52df4c2ef43","Type":"ContainerStarted","Data":"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299"} Sep 14 05:01:56.779710 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.779590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" event={"ID":"41de4b1f-5003-4dd7-a72f-e52df4c2ef43","Type":"ContainerStarted","Data":"0b664c648e80562d30c1e4d019ca2a6160e6c5424e7f9e6a3d3d3253ac03f835"} Sep 14 05:01:56.780988 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.780935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:01:56.801373 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.801298 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" podStartSLOduration=1.801276106 podStartE2EDuration="1.801276106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:56.797005075 +0000 UTC m=+927.754621813" watchObservedRunningTime="2026-09-14 05:01:56.801276106 +0000 UTC m=+927.758892843" Sep 14 05:01:56.888082 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.888041 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:56.888399 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:56.888351 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:57.756055 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:57.755623 2581 patch_prober.go:28] interesting pod/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 14 05:01:57.756055 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:57.755687 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 14 05:01:57.887527 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:57.887489 2581 patch_prober.go:28] interesting pod/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 14 05:01:57.888088 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:57.888054 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 14 05:01:58.760646 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.760613 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:58.760931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.760905 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:58.761548 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.761533 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:01:58.891719 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.891684 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:58.892153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.891969 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:01:58.892606 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:01:58.892590 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:02:01.809960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.809861 2581 generic.go:356] "Generic (PLEG): container finished" podID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerID="786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60" exitCode=0 Sep 14 05:02:01.809960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.809938 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerDied","Data":"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60"} Sep 14 05:02:01.810423 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.810386 2581 scope.go:117] "RemoveContainer" containerID="786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60" Sep 14 05:02:01.811499 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.811474 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:01.811568 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.811555 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:01.811613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:01.811592 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:02.817360 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:02.817315 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerStarted","Data":"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151"} Sep 14 05:02:03.446498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:03.446463 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:02:03.446893 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:03.446808 2581 patch_prober.go:28] interesting pod/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 14 05:02:03.447076 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:03.447041 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 14 05:02:04.446809 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:04.446771 2581 patch_prober.go:28] interesting pod/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 14 05:02:04.447420 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:04.447389 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 14 05:02:05.446384 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:05.446339 2581 patch_prober.go:28] interesting pod/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 14 05:02:05.446596 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:05.446427 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 14 05:02:06.446201 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:06.446156 2581 patch_prober.go:28] interesting pod/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 14 05:02:06.446696 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:06.446229 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 14 05:02:07.448987 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:07.448555 2581 patch_prober.go:28] interesting pod/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 14 05:02:07.448987 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:07.448618 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 14 05:02:07.671141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:07.668170 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:02:07.679989 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:07.677328 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" containerID="cri-o://906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151" gracePeriod=30 Sep 14 05:02:08.631770 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.631725 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:08.637252 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.637215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.640174 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.640147 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-60e333a9-gw-openshift-default-dockercfg-bqclx\"" Sep 14 05:02:08.661962 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.661925 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:08.668150 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4gx\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668325 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668189 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668325 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668280 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668535 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668585 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668585 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.668681 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.668595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.771836 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772020 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772020 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772020 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772020 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772020 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.771969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772412 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.772040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4gx\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772412 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.772074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772412 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.772182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.772960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.772932 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.773413 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.773392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.775819 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.775484 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.775819 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.775769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.775819 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.775771 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.777618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.777572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.781320 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.781293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.784259 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.784209 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.785697 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.785668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4gx\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx\") pod \"e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:08.796271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.796243 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:02:08.949393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:08.949314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:09.235635 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.235608 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:09.237816 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:02:09.237789 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61c3a85_c0a8_43f6_827a_55709eb1d0fe.slice/crio-cea4da612dc1012b3f80e446273bb8e99acbb79789647718c590b3a29a4cf825 WatchSource:0}: Error finding container cea4da612dc1012b3f80e446273bb8e99acbb79789647718c590b3a29a4cf825: Status 404 returned error can't find the container with id cea4da612dc1012b3f80e446273bb8e99acbb79789647718c590b3a29a4cf825 Sep 14 05:02:09.240860 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.240824 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:09.240935 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.240906 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:09.240971 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.240947 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:09.874665 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.874631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerStarted","Data":"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c"} Sep 14 05:02:09.875049 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.874682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerStarted","Data":"cea4da612dc1012b3f80e446273bb8e99acbb79789647718c590b3a29a4cf825"} Sep 14 05:02:09.893295 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.893225 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podStartSLOduration=1.893205545 podStartE2EDuration="1.893205545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:09.892183692 +0000 UTC m=+940.849800428" watchObservedRunningTime="2026-09-14 05:02:09.893205545 +0000 UTC m=+940.850822283" Sep 14 05:02:09.950244 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:09.950215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:10.601318 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.601214 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:02:10.881968 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.881888 2581 generic.go:356] "Generic (PLEG): container finished" podID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerID="b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b" exitCode=0 Sep 14 05:02:10.881968 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.881959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerDied","Data":"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b"} Sep 14 05:02:10.882482 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.882466 2581 scope.go:117] "RemoveContainer" containerID="b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b" Sep 14 05:02:10.883544 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.883521 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:10.883601 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.883586 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:10.883642 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.883620 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:10.955540 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:10.955514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:11.889155 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:11.889097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerStarted","Data":"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed"} Sep 14 05:02:11.889532 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:11.889368 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:11.890382 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:11.890363 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:12.319144 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:12.319039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:02:12.319328 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:12.319275 2581 patch_prober.go:28] interesting pod/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 05:02:12.319407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:12.319341 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 05:02:13.318881 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.318845 2581 patch_prober.go:28] interesting pod/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 05:02:13.319285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.318912 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 05:02:13.884216 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.884192 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:02:13.906388 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.906354 2581 generic.go:356] "Generic (PLEG): container finished" podID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerID="906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151" exitCode=0 Sep 14 05:02:13.906571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.906417 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" Sep 14 05:02:13.906571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.906501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerDied","Data":"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151"} Sep 14 05:02:13.906571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.906550 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t" event={"ID":"95706d1e-36cb-4ceb-95d5-a2d34f453439","Type":"ContainerDied","Data":"ed623f670ecc35bf3801271eb7234586652cc733540a9f886381658d1a92a197"} Sep 14 05:02:13.906754 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.906597 2581 scope.go:117] "RemoveContainer" containerID="906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151" Sep 14 05:02:13.916240 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.916220 2581 scope.go:117] "RemoveContainer" containerID="786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60" Sep 14 05:02:13.923424 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923405 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwbs8\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923435 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923480 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923649 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923542 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923649 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923580 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923649 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923604 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923703 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923726 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923756 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket\") pod \"95706d1e-36cb-4ceb-95d5-a2d34f453439\" (UID: \"95706d1e-36cb-4ceb-95d5-a2d34f453439\") " Sep 14 05:02:13.923806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923768 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:13.924019 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923848 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:13.924019 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.923985 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data" (OuterVolumeSpecName: "istio-data") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:13.924241 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.924189 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:13.924241 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.924209 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:13.924241 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.924217 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:13.924478 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.924250 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:13.924478 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.924108 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:02:13.926415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.926284 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:02:13.926415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.926375 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8" (OuterVolumeSpecName: "kube-api-access-fwbs8") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "kube-api-access-fwbs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:02:13.926553 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.926445 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:13.926553 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.926462 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token" (OuterVolumeSpecName: "istio-token") pod "95706d1e-36cb-4ceb-95d5-a2d34f453439" (UID: "95706d1e-36cb-4ceb-95d5-a2d34f453439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:02:13.926840 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.926820 2581 scope.go:117] "RemoveContainer" containerID="906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151" Sep 14 05:02:13.927150 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:02:13.927102 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151\": container with ID starting with 906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151 not found: ID does not exist" containerID="906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151" Sep 14 05:02:13.927248 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.927161 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151"} err="failed to get container status \"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151\": rpc error: code = NotFound desc = could not find container \"906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151\": container with ID starting with 906ca0a74d49566e2b52c8bba1c0384116cb2dcdc00adf5ae16c506ffc639151 not found: ID does not exist" Sep 14 05:02:13.927248 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.927184 2581 scope.go:117] "RemoveContainer" containerID="786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60" Sep 14 05:02:13.927492 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:02:13.927471 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60\": container with ID starting with 786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60 not found: ID does not exist" containerID="786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60" Sep 14 05:02:13.927553 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:13.927499 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60"} err="failed to get container status \"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60\": rpc error: code = NotFound desc = could not find container \"786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60\": container with ID starting with 786c361630e9839fbe6b6097045c167a685eea1c635c6ac4f570e7400d2a9f60 not found: ID does not exist" Sep 14 05:02:14.025390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025359 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbs8\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-kube-api-access-fwbs8\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.025390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025387 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.025390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025401 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95706d1e-36cb-4ceb-95d5-a2d34f453439-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.025667 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025413 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.025667 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025426 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95706d1e-36cb-4ceb-95d5-a2d34f453439-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.025667 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.025438 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95706d1e-36cb-4ceb-95d5-a2d34f453439-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:14.227644 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.227613 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:02:14.231803 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.231780 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-654e8bde-openshift-default-6dbc8bc8-bh59t"] Sep 14 05:02:14.318710 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.318671 2581 patch_prober.go:28] interesting pod/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 05:02:14.318877 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:14.318728 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 05:02:15.319196 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:15.319155 2581 patch_prober.go:28] interesting pod/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 05:02:15.319571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:15.319218 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 05:02:15.557985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:15.557944 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" path="/var/lib/kubelet/pods/95706d1e-36cb-4ceb-95d5-a2d34f453439/volumes" Sep 14 05:02:16.319372 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:16.319333 2581 patch_prober.go:28] interesting pod/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 14 05:02:16.319758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:16.319396 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 14 05:02:17.323254 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:17.323227 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:02:17.323636 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:17.323484 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:02:17.324275 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:17.324252 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:02:31.008761 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.008729 2581 generic.go:356] "Generic (PLEG): container finished" podID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerID="353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c" exitCode=0 Sep 14 05:02:31.009228 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.008801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerDied","Data":"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c"} Sep 14 05:02:31.009278 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.009231 2581 scope.go:117] "RemoveContainer" containerID="353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c" Sep 14 05:02:31.010362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.010332 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:31.010474 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.010414 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:31.010474 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:31.010462 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:02:32.015735 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:32.015703 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerStarted","Data":"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610"} Sep 14 05:02:32.950489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:32.950444 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:32.951013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:32.950984 2581 patch_prober.go:28] interesting pod/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 14 05:02:32.951152 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:32.951079 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 14 05:02:33.950529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:33.950489 2581 patch_prober.go:28] interesting pod/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 14 05:02:33.951100 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:33.951067 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 14 05:02:34.949889 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:34.949847 2581 patch_prober.go:28] interesting pod/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 14 05:02:34.950147 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:34.949937 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 14 05:02:35.950582 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:35.950537 2581 patch_prober.go:28] interesting pod/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 14 05:02:35.951070 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:35.950611 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 14 05:02:36.950054 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:36.950007 2581 patch_prober.go:28] interesting pod/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 14 05:02:36.950294 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:36.950083 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 14 05:02:37.954065 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:37.954031 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:37.954495 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:37.954349 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:37.955096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:37.955075 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:43.558179 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:43.558146 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:43.558776 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:43.558360 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" containerID="cri-o://9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610" gracePeriod=30 Sep 14 05:02:48.828763 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.828696 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:48.908962 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.908929 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w4gx\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.908962 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.908967 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909004 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909087 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909156 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909194 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909234 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909256 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909354 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:48.909489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909423 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:48.909657 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909489 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy\") pod \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\" (UID: \"a61c3a85-c0a8-43f6-827a-55709eb1d0fe\") " Sep 14 05:02:48.909657 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909508 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:48.909657 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909523 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:48.909657 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909613 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:02:48.909883 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909858 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:48.909960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909886 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:48.909960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909900 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:48.909960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909909 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:48.909960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.909916 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:48.911401 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.911382 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:02:48.911493 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.911475 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:02:48.911705 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.911684 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx" (OuterVolumeSpecName: "kube-api-access-7w4gx") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "kube-api-access-7w4gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:02:48.911765 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:48.911693 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a61c3a85-c0a8-43f6-827a-55709eb1d0fe" (UID: "a61c3a85-c0a8-43f6-827a-55709eb1d0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:02:49.010497 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.010460 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:49.010497 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.010490 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:49.010497 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.010500 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4gx\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-kube-api-access-7w4gx\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:49.010738 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.010509 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61c3a85-c0a8-43f6-827a-55709eb1d0fe-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:02:49.108226 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.108192 2581 generic.go:356] "Generic (PLEG): container finished" podID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerID="9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610" exitCode=0 Sep 14 05:02:49.108394 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.108247 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" Sep 14 05:02:49.108394 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.108275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerDied","Data":"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610"} Sep 14 05:02:49.108394 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.108353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm" event={"ID":"a61c3a85-c0a8-43f6-827a-55709eb1d0fe","Type":"ContainerDied","Data":"cea4da612dc1012b3f80e446273bb8e99acbb79789647718c590b3a29a4cf825"} Sep 14 05:02:49.108394 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.108375 2581 scope.go:117] "RemoveContainer" containerID="9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610" Sep 14 05:02:49.118416 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.118399 2581 scope.go:117] "RemoveContainer" containerID="353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c" Sep 14 05:02:49.126420 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.126378 2581 scope.go:117] "RemoveContainer" containerID="9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610" Sep 14 05:02:49.126902 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:02:49.126834 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610\": container with ID starting with 9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610 not found: ID does not exist" containerID="9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610" Sep 14 05:02:49.127006 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.126902 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610"} err="failed to get container status \"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610\": rpc error: code = NotFound desc = could not find container \"9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610\": container with ID starting with 9c2f5d57c33bd391d7b3e4ab6a23f3b20820e74e50c9986585cbc12c70a3b610 not found: ID does not exist" Sep 14 05:02:49.127006 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.126927 2581 scope.go:117] "RemoveContainer" containerID="353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c" Sep 14 05:02:49.127352 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:02:49.127321 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c\": container with ID starting with 353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c not found: ID does not exist" containerID="353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c" Sep 14 05:02:49.127456 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.127363 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c"} err="failed to get container status \"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c\": rpc error: code = NotFound desc = could not find container \"353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c\": container with ID starting with 353cb28ad63c764de298f9d78699641d71fe4f1ba44e8c82961c7da5c528b69c not found: ID does not exist" Sep 14 05:02:49.129348 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.129327 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:49.135263 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.135242 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-60e333a9-gw-openshift-default-5496f8f78d-7dwrm"] Sep 14 05:02:49.524666 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.524568 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:02:49.525878 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.525851 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:02:49.526032 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526020 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:02:49.526170 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526158 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.526260 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526250 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.526350 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526340 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.526430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526421 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.526706 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526693 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:02:49.526803 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526794 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.526876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526867 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:02:49.526960 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.526951 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95706d1e-36cb-4ceb-95d5-a2d34f453439" containerName="istio-proxy" Sep 14 05:02:49.533558 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.533529 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.535504 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.535478 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8f579f00\"" Sep 14 05:02:49.535625 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.535534 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8f579f00-dockercfg-mklc2\"" Sep 14 05:02:49.535625 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.535605 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:02:49.556078 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.556044 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" path="/var/lib/kubelet/pods/a61c3a85-c0a8-43f6-827a-55709eb1d0fe/volumes" Sep 14 05:02:49.616364 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.616320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.616364 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.616364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24cmq\" (UniqueName: \"kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.717148 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.717088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.717340 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.717161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24cmq\" (UniqueName: \"kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.717848 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.717826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.723932 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.723903 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24cmq\" (UniqueName: \"kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq\") pod \"payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:49.847845 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:49.847767 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:50.128701 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:50.128660 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:02:50.130905 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:02:50.130883 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54af0562_f64c_4cdf_843f_d1cb70bef744.slice/crio-889dd4f9d32916bf33ac500703d48e9f32bb5968dacccb7789d2e800611afb92 WatchSource:0}: Error finding container 889dd4f9d32916bf33ac500703d48e9f32bb5968dacccb7789d2e800611afb92: Status 404 returned error can't find the container with id 889dd4f9d32916bf33ac500703d48e9f32bb5968dacccb7789d2e800611afb92 Sep 14 05:02:51.121615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:51.121580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" event={"ID":"54af0562-f64c-4cdf-843f-d1cb70bef744","Type":"ContainerStarted","Data":"75c71b8bafcde08094e75dd4d4622f371cd944ff6b575c07a3108183fd1e019d"} Sep 14 05:02:51.121965 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:51.121633 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" event={"ID":"54af0562-f64c-4cdf-843f-d1cb70bef744","Type":"ContainerStarted","Data":"889dd4f9d32916bf33ac500703d48e9f32bb5968dacccb7789d2e800611afb92"} Sep 14 05:02:51.121965 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:51.121696 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:02:51.138753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:02:51.138707 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" podStartSLOduration=2.13869375 podStartE2EDuration="2.13869375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:51.136916824 +0000 UTC m=+982.094533560" watchObservedRunningTime="2026-09-14 05:02:51.13869375 +0000 UTC m=+982.096310511" Sep 14 05:03:02.129264 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:02.129189 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:03:12.783529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.783497 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g"] Sep 14 05:03:12.783936 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.783921 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:03:12.783998 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.783939 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61c3a85-c0a8-43f6-827a-55709eb1d0fe" containerName="istio-proxy" Sep 14 05:03:12.787335 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.787319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:12.789273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.789255 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:03:12.789367 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.789255 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:03:12.795013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.794762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g"] Sep 14 05:03:12.831375 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.831351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g\" (UID: \"a3743ffe-5cf8-4ddc-a36e-342925e47760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:12.932526 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.932503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g\" (UID: \"a3743ffe-5cf8-4ddc-a36e-342925e47760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:12.933085 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:12.933066 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g\" (UID: \"a3743ffe-5cf8-4ddc-a36e-342925e47760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:13.098916 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:13.098850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:13.370694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:13.370668 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g"] Sep 14 05:03:13.372547 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:03:13.372517 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3743ffe_5cf8_4ddc_a36e_342925e47760.slice/crio-fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c WatchSource:0}: Error finding container fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c: Status 404 returned error can't find the container with id fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c Sep 14 05:03:14.246225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:14.246188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" event={"ID":"a3743ffe-5cf8-4ddc-a36e-342925e47760","Type":"ContainerStarted","Data":"fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c"} Sep 14 05:03:17.266820 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:17.266781 2581 generic.go:356] "Generic (PLEG): container finished" podID="a3743ffe-5cf8-4ddc-a36e-342925e47760" containerID="b4385e7e6265c4023310d24ba0d3ba2156c1dcea16a9770229e95d1a330c5ece" exitCode=0 Sep 14 05:03:17.267243 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:17.266870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" event={"ID":"a3743ffe-5cf8-4ddc-a36e-342925e47760","Type":"ContainerDied","Data":"b4385e7e6265c4023310d24ba0d3ba2156c1dcea16a9770229e95d1a330c5ece"} Sep 14 05:03:18.433915 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:18.433891 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:18.489510 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:18.489485 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca\") pod \"a3743ffe-5cf8-4ddc-a36e-342925e47760\" (UID: \"a3743ffe-5cf8-4ddc-a36e-342925e47760\") " Sep 14 05:03:18.489811 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:18.489788 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3743ffe-5cf8-4ddc-a36e-342925e47760" (UID: "a3743ffe-5cf8-4ddc-a36e-342925e47760"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:03:18.489894 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:18.489862 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3743ffe-5cf8-4ddc-a36e-342925e47760-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:03:19.279096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:19.279069 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" Sep 14 05:03:19.279309 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:19.279110 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g" event={"ID":"a3743ffe-5cf8-4ddc-a36e-342925e47760","Type":"ContainerDied","Data":"fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c"} Sep 14 05:03:19.279309 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:19.279172 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe160e83f3b639d81b1ac824fd261b99db1851004bfb88a8718bcae2cc8a614c" Sep 14 05:03:29.062200 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.062137 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:03:29.062712 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.062408 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" podUID="54af0562-f64c-4cdf-843f-d1cb70bef744" containerName="payload-pre-processing" containerID="cri-o://75c71b8bafcde08094e75dd4d4622f371cd944ff6b575c07a3108183fd1e019d" gracePeriod=30 Sep 14 05:03:29.216309 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.216272 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g"] Sep 14 05:03:29.218524 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.218498 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f579f00-x4z8g"] Sep 14 05:03:29.344016 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.343988 2581 generic.go:356] "Generic (PLEG): container finished" podID="54af0562-f64c-4cdf-843f-d1cb70bef744" containerID="75c71b8bafcde08094e75dd4d4622f371cd944ff6b575c07a3108183fd1e019d" exitCode=0 Sep 14 05:03:29.344174 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.344157 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" event={"ID":"54af0562-f64c-4cdf-843f-d1cb70bef744","Type":"ContainerDied","Data":"75c71b8bafcde08094e75dd4d4622f371cd944ff6b575c07a3108183fd1e019d"} Sep 14 05:03:29.370198 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.370177 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:03:29.500455 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.500426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume\") pod \"54af0562-f64c-4cdf-843f-d1cb70bef744\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " Sep 14 05:03:29.500455 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.500464 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24cmq\" (UniqueName: \"kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq\") pod \"54af0562-f64c-4cdf-843f-d1cb70bef744\" (UID: \"54af0562-f64c-4cdf-843f-d1cb70bef744\") " Sep 14 05:03:29.500854 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.500832 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54af0562-f64c-4cdf-843f-d1cb70bef744" (UID: "54af0562-f64c-4cdf-843f-d1cb70bef744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:03:29.502515 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.502488 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq" (OuterVolumeSpecName: "kube-api-access-24cmq") pod "54af0562-f64c-4cdf-843f-d1cb70bef744" (UID: "54af0562-f64c-4cdf-843f-d1cb70bef744"). InnerVolumeSpecName "kube-api-access-24cmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:03:29.553357 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.553328 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3743ffe-5cf8-4ddc-a36e-342925e47760" path="/var/lib/kubelet/pods/a3743ffe-5cf8-4ddc-a36e-342925e47760/volumes" Sep 14 05:03:29.601523 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.601499 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24cmq\" (UniqueName: \"kubernetes.io/projected/54af0562-f64c-4cdf-843f-d1cb70bef744-kube-api-access-24cmq\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:03:29.601617 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:29.601529 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54af0562-f64c-4cdf-843f-d1cb70bef744-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:03:30.351148 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:30.351103 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" Sep 14 05:03:30.351579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:30.351147 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797" event={"ID":"54af0562-f64c-4cdf-843f-d1cb70bef744","Type":"ContainerDied","Data":"889dd4f9d32916bf33ac500703d48e9f32bb5968dacccb7789d2e800611afb92"} Sep 14 05:03:30.351579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:30.351241 2581 scope.go:117] "RemoveContainer" containerID="75c71b8bafcde08094e75dd4d4622f371cd944ff6b575c07a3108183fd1e019d" Sep 14 05:03:30.369879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:30.369853 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:03:30.375610 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:30.375584 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f579f00-848cdc564b-k8797"] Sep 14 05:03:31.237377 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.237345 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:03:31.237931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.237908 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3743ffe-5cf8-4ddc-a36e-342925e47760" containerName="revoke-keys" Sep 14 05:03:31.237931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.237931 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3743ffe-5cf8-4ddc-a36e-342925e47760" containerName="revoke-keys" Sep 14 05:03:31.238101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.237971 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54af0562-f64c-4cdf-843f-d1cb70bef744" containerName="payload-pre-processing" Sep 14 05:03:31.238101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.237978 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54af0562-f64c-4cdf-843f-d1cb70bef744" containerName="payload-pre-processing" Sep 14 05:03:31.238101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.238054 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54af0562-f64c-4cdf-843f-d1cb70bef744" containerName="payload-pre-processing" Sep 14 05:03:31.238101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.238068 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3743ffe-5cf8-4ddc-a36e-342925e47760" containerName="revoke-keys" Sep 14 05:03:31.241236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.241215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.242952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.242932 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2a2a3973-openshift-default-dockercfg-brqc8\"" Sep 14 05:03:31.253752 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.253726 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:03:31.319679 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.319679 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr46h\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.319953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.319953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.319953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.319953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.320220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.319978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.320220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.320032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.320220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.320084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.420723 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.420961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr46h\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421206 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.421753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.421708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.423222 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.423198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.423330 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.423312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.427988 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.427953 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.428073 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.428024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr46h\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h\") pod \"e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.553495 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.553414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:31.553635 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.553576 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54af0562-f64c-4cdf-843f-d1cb70bef744" path="/var/lib/kubelet/pods/54af0562-f64c-4cdf-843f-d1cb70bef744/volumes" Sep 14 05:03:31.835858 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.835832 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:03:31.837161 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:03:31.837135 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12164864_c838_4c83_b7d8_f9ab8fb50768.slice/crio-7eb48314ea26976ab14784e8d01ec6c9baf93797e5a9609e5e4f6aa7f5794e15 WatchSource:0}: Error finding container 7eb48314ea26976ab14784e8d01ec6c9baf93797e5a9609e5e4f6aa7f5794e15: Status 404 returned error can't find the container with id 7eb48314ea26976ab14784e8d01ec6c9baf93797e5a9609e5e4f6aa7f5794e15 Sep 14 05:03:31.839698 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.839665 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:31.839814 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.839739 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:31.839814 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:31.839780 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:32.368172 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:32.368112 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerStarted","Data":"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d"} Sep 14 05:03:32.368342 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:32.368198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerStarted","Data":"7eb48314ea26976ab14784e8d01ec6c9baf93797e5a9609e5e4f6aa7f5794e15"} Sep 14 05:03:32.387181 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:32.387107 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podStartSLOduration=1.387088382 podStartE2EDuration="1.387088382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:03:32.386544086 +0000 UTC m=+1023.344160823" watchObservedRunningTime="2026-09-14 05:03:32.387088382 +0000 UTC m=+1023.344705120" Sep 14 05:03:32.553571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:32.553543 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:32.558236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:32.558215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:33.374066 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:33.374034 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:33.374837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:33.374814 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:56.507377 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.507340 2581 generic.go:356] "Generic (PLEG): container finished" podID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerID="46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d" exitCode=0 Sep 14 05:03:56.507781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.507411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerDied","Data":"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d"} Sep 14 05:03:56.507829 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.507808 2581 scope.go:117] "RemoveContainer" containerID="46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d" Sep 14 05:03:56.508939 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.508911 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:56.509043 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.508996 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:56.509107 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:56.509053 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:03:57.515818 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:57.515782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerStarted","Data":"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0"} Sep 14 05:03:57.553845 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:57.553813 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:03:57.554021 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:57.553998 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:03:57.554072 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:57.554044 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:03:58.554621 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:58.554571 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:03:58.555168 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:58.554646 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:03:59.553942 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:59.553903 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:03:59.554164 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:03:59.553967 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:04:00.554042 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:00.553999 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:04:00.554541 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:00.554078 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:04:01.557236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:01.557200 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:04:01.557819 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:01.557789 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:04:02.554404 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:02.554367 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:04:02.554681 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:02.554653 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:04:03.374761 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.374722 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:04:03.553813 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.553772 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 14 05:04:03.553995 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.553828 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 14 05:04:03.792629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.792532 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5"] Sep 14 05:04:03.796202 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.796181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:03.797844 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.797817 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:04:03.797947 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.797851 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:04:03.802919 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.802895 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5"] Sep 14 05:04:03.971905 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:03.971870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5\" (UID: \"7cae533d-2804-43ba-b337-f33d1d0ef8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:04.072630 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.072555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5\" (UID: \"7cae533d-2804-43ba-b337-f33d1d0ef8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:04.073210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.073191 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5\" (UID: \"7cae533d-2804-43ba-b337-f33d1d0ef8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:04.108992 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.108971 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:04.381688 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.381658 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5"] Sep 14 05:04:04.384325 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:04:04.384295 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cae533d_2804_43ba_b337_f33d1d0ef8a5.slice/crio-e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789 WatchSource:0}: Error finding container e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789: Status 404 returned error can't find the container with id e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789 Sep 14 05:04:04.558466 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.558443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:04:04.559681 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.559660 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:04:04.560167 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:04.560139 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" event={"ID":"7cae533d-2804-43ba-b337-f33d1d0ef8a5","Type":"ContainerStarted","Data":"e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789"} Sep 14 05:04:05.566873 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:05.566840 2581 generic.go:356] "Generic (PLEG): container finished" podID="7cae533d-2804-43ba-b337-f33d1d0ef8a5" containerID="a35acad051d5a868081dd71f490bd3583530d95ce77437768bc7ed591ca1e648" exitCode=0 Sep 14 05:04:05.567249 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:05.566928 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" event={"ID":"7cae533d-2804-43ba-b337-f33d1d0ef8a5","Type":"ContainerDied","Data":"a35acad051d5a868081dd71f490bd3583530d95ce77437768bc7ed591ca1e648"} Sep 14 05:04:06.717734 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:06.717712 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:06.907658 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:06.907631 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca\") pod \"7cae533d-2804-43ba-b337-f33d1d0ef8a5\" (UID: \"7cae533d-2804-43ba-b337-f33d1d0ef8a5\") " Sep 14 05:04:06.907973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:06.907950 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cae533d-2804-43ba-b337-f33d1d0ef8a5" (UID: "7cae533d-2804-43ba-b337-f33d1d0ef8a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:04:07.009076 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:07.009029 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae533d-2804-43ba-b337-f33d1d0ef8a5-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:07.578581 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:07.578552 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" Sep 14 05:04:07.578581 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:07.578577 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5" event={"ID":"7cae533d-2804-43ba-b337-f33d1d0ef8a5","Type":"ContainerDied","Data":"e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789"} Sep 14 05:04:07.578756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:07.578613 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e826e81bc0255a07e242d5be8cbc56d995e79c5ebe368de0c25e24ef05d789" Sep 14 05:04:15.956560 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:15.956520 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5"] Sep 14 05:04:15.958702 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:15.958679 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2a2a3973-flwg5"] Sep 14 05:04:16.109402 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:16.109365 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:04:16.109872 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:16.109842 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" containerID="cri-o://111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0" gracePeriod=30 Sep 14 05:04:17.553386 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:17.553354 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cae533d-2804-43ba-b337-f33d1d0ef8a5" path="/var/lib/kubelet/pods/7cae533d-2804-43ba-b337-f33d1d0ef8a5/volumes" Sep 14 05:04:19.561191 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:19.561159 2581 patch_prober.go:28] interesting pod/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 05:04:19.561586 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:19.561219 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 05:04:21.368378 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.368357 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:04:21.451992 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.451916 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.451992 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.451956 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452004 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452064 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr46h\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452079 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452163 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452191 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452221 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:04:21.452591 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert\") pod \"12164864-c838-4c83-b7d8-f9ab8fb50768\" (UID: \"12164864-c838-4c83-b7d8-f9ab8fb50768\") " Sep 14 05:04:21.452591 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452327 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:04:21.452591 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data" (OuterVolumeSpecName: "istio-data") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:04:21.452591 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452451 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:04:21.452794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452685 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:04:21.452794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452756 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12164864-c838-4c83-b7d8-f9ab8fb50768-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.452794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452774 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.452794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452784 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.452794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452792 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.453061 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.452801 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.454322 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.454298 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h" (OuterVolumeSpecName: "kube-api-access-zr46h") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "kube-api-access-zr46h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:04:21.454454 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.454394 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:04:21.454708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.454687 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:04:21.454708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.454698 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token" (OuterVolumeSpecName: "istio-token") pod "12164864-c838-4c83-b7d8-f9ab8fb50768" (UID: "12164864-c838-4c83-b7d8-f9ab8fb50768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:04:21.554924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.554052 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr46h\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-kube-api-access-zr46h\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.554924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.554094 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.554924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.554108 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.554924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.554139 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12164864-c838-4c83-b7d8-f9ab8fb50768-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:21.652349 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.652319 2581 generic.go:356] "Generic (PLEG): container finished" podID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerID="111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0" exitCode=0 Sep 14 05:04:21.652543 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.652378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerDied","Data":"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0"} Sep 14 05:04:21.652543 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.652384 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" Sep 14 05:04:21.652543 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.652441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx" event={"ID":"12164864-c838-4c83-b7d8-f9ab8fb50768","Type":"ContainerDied","Data":"7eb48314ea26976ab14784e8d01ec6c9baf93797e5a9609e5e4f6aa7f5794e15"} Sep 14 05:04:21.652543 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.652463 2581 scope.go:117] "RemoveContainer" containerID="111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0" Sep 14 05:04:21.661497 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.661476 2581 scope.go:117] "RemoveContainer" containerID="46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d" Sep 14 05:04:21.669787 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.669768 2581 scope.go:117] "RemoveContainer" containerID="111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0" Sep 14 05:04:21.670046 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:04:21.670026 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0\": container with ID starting with 111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0 not found: ID does not exist" containerID="111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0" Sep 14 05:04:21.670152 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.670058 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0"} err="failed to get container status \"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0\": rpc error: code = NotFound desc = could not find container \"111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0\": container with ID starting with 111402a37183cea7e0d489dafa3ba5d22cfa5ff58b7e24e839623ab9e2f216b0 not found: ID does not exist" Sep 14 05:04:21.670152 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.670083 2581 scope.go:117] "RemoveContainer" containerID="46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d" Sep 14 05:04:21.670236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.670173 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:04:21.670361 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:04:21.670345 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d\": container with ID starting with 46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d not found: ID does not exist" containerID="46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d" Sep 14 05:04:21.670395 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.670366 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d"} err="failed to get container status \"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d\": rpc error: code = NotFound desc = could not find container \"46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d\": container with ID starting with 46c1ec0d097cf4073df856dec0a7937b7a8ca5f6221cfb38e4f1b1848909265d not found: ID does not exist" Sep 14 05:04:21.674313 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:21.674293 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2a2a3973-openshift-default-7698ff55bd-8mfjx"] Sep 14 05:04:23.552898 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:23.552823 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" path="/var/lib/kubelet/pods/12164864-c838-4c83-b7d8-f9ab8fb50768/volumes" Sep 14 05:04:49.830943 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.830909 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5"] Sep 14 05:04:49.831405 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831364 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cae533d-2804-43ba-b337-f33d1d0ef8a5" containerName="revoke-keys" Sep 14 05:04:49.831405 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831376 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cae533d-2804-43ba-b337-f33d1d0ef8a5" containerName="revoke-keys" Sep 14 05:04:49.831405 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831403 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:04:49.831529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831409 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:04:49.831529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831482 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:04:49.831529 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.831495 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cae533d-2804-43ba-b337-f33d1d0ef8a5" containerName="revoke-keys" Sep 14 05:04:49.836253 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.836234 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:49.838414 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.838390 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:04:49.838520 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.838438 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:04:49.842799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.842775 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5"] Sep 14 05:04:49.924428 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:49.924391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f14475-zljs5\" (UID: \"55f0563a-d6f8-4169-bf33-c1a63281fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:50.025711 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.025676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f14475-zljs5\" (UID: \"55f0563a-d6f8-4169-bf33-c1a63281fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:50.026321 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.026299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f14475-zljs5\" (UID: \"55f0563a-d6f8-4169-bf33-c1a63281fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:50.147917 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.147883 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:50.421064 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.421024 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5"] Sep 14 05:04:50.422622 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:04:50.422597 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f0563a_d6f8_4169_bf33_c1a63281fca4.slice/crio-45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a WatchSource:0}: Error finding container 45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a: Status 404 returned error can't find the container with id 45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a Sep 14 05:04:50.807816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.807783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" event={"ID":"55f0563a-d6f8-4169-bf33-c1a63281fca4","Type":"ContainerStarted","Data":"1ec314d064c803f3e1d98556fc894f7052f34459ce7ead560fc47d6a44956a9c"} Sep 14 05:04:50.807996 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:50.807843 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" event={"ID":"55f0563a-d6f8-4169-bf33-c1a63281fca4","Type":"ContainerStarted","Data":"45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a"} Sep 14 05:04:51.814756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:51.814722 2581 generic.go:356] "Generic (PLEG): container finished" podID="55f0563a-d6f8-4169-bf33-c1a63281fca4" containerID="1ec314d064c803f3e1d98556fc894f7052f34459ce7ead560fc47d6a44956a9c" exitCode=0 Sep 14 05:04:51.815238 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:51.814765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" event={"ID":"55f0563a-d6f8-4169-bf33-c1a63281fca4","Type":"ContainerDied","Data":"1ec314d064c803f3e1d98556fc894f7052f34459ce7ead560fc47d6a44956a9c"} Sep 14 05:04:52.967409 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:52.967381 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:53.058615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.058574 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca\") pod \"55f0563a-d6f8-4169-bf33-c1a63281fca4\" (UID: \"55f0563a-d6f8-4169-bf33-c1a63281fca4\") " Sep 14 05:04:53.059034 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.059003 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55f0563a-d6f8-4169-bf33-c1a63281fca4" (UID: "55f0563a-d6f8-4169-bf33-c1a63281fca4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:04:53.160172 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.160103 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f0563a-d6f8-4169-bf33-c1a63281fca4-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:04:53.827876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.827788 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" event={"ID":"55f0563a-d6f8-4169-bf33-c1a63281fca4","Type":"ContainerDied","Data":"45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a"} Sep 14 05:04:53.827876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.827814 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5" Sep 14 05:04:53.827876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:04:53.827838 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45477b668134d35280a3d7ce1ab3b0be74f5cd6d1418fa71846c16c72d48a37a" Sep 14 05:05:00.320193 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320158 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw"] Sep 14 05:05:00.320590 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320578 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f0563a-d6f8-4169-bf33-c1a63281fca4" containerName="revoke-keys" Sep 14 05:05:00.320634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320591 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f0563a-d6f8-4169-bf33-c1a63281fca4" containerName="revoke-keys" Sep 14 05:05:00.320634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320606 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:05:00.320634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320612 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:05:00.320730 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320686 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12164864-c838-4c83-b7d8-f9ab8fb50768" containerName="istio-proxy" Sep 14 05:05:00.320730 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.320694 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f0563a-d6f8-4169-bf33-c1a63281fca4" containerName="revoke-keys" Sep 14 05:05:00.323756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.323741 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:00.325527 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.325506 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:05:00.325662 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.325524 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:05:00.330663 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.330636 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw"] Sep 14 05:05:00.424286 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.424257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw\" (UID: \"28714f34-228b-400a-be2f-48015a3258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:00.525303 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.525274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw\" (UID: \"28714f34-228b-400a-be2f-48015a3258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:00.525879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.525860 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw\" (UID: \"28714f34-228b-400a-be2f-48015a3258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:00.635001 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.634965 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:00.907049 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:00.907019 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw"] Sep 14 05:05:00.909033 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:05:00.909002 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28714f34_228b_400a_be2f_48015a3258c4.slice/crio-d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd WatchSource:0}: Error finding container d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd: Status 404 returned error can't find the container with id d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd Sep 14 05:05:01.840824 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:01.840788 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5"] Sep 14 05:05:01.845480 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:01.845454 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f14475-zljs5"] Sep 14 05:05:01.869344 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:01.869317 2581 generic.go:356] "Generic (PLEG): container finished" podID="28714f34-228b-400a-be2f-48015a3258c4" containerID="a9bb33aa8e4e493930dc6d86a82982b86b180039c4e3d25e9a4da36888de8e95" exitCode=0 Sep 14 05:05:01.869461 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:01.869381 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" event={"ID":"28714f34-228b-400a-be2f-48015a3258c4","Type":"ContainerDied","Data":"a9bb33aa8e4e493930dc6d86a82982b86b180039c4e3d25e9a4da36888de8e95"} Sep 14 05:05:01.869461 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:01.869421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" event={"ID":"28714f34-228b-400a-be2f-48015a3258c4","Type":"ContainerStarted","Data":"d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd"} Sep 14 05:05:03.007287 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.007262 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:03.049746 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.049716 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca\") pod \"28714f34-228b-400a-be2f-48015a3258c4\" (UID: \"28714f34-228b-400a-be2f-48015a3258c4\") " Sep 14 05:05:03.050073 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.050050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28714f34-228b-400a-be2f-48015a3258c4" (UID: "28714f34-228b-400a-be2f-48015a3258c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:05:03.050256 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.050239 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28714f34-228b-400a-be2f-48015a3258c4-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:05:03.553431 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.553399 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f0563a-d6f8-4169-bf33-c1a63281fca4" path="/var/lib/kubelet/pods/55f0563a-d6f8-4169-bf33-c1a63281fca4/volumes" Sep 14 05:05:03.881008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.880980 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" Sep 14 05:05:03.881196 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.881026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw" event={"ID":"28714f34-228b-400a-be2f-48015a3258c4","Type":"ContainerDied","Data":"d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd"} Sep 14 05:05:03.881196 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:03.881073 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d151987a9e9b8b37b85d62817daaf47aab5efc5b0737da46ab50c5ebd24378bd" Sep 14 05:05:11.343545 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:11.343508 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw"] Sep 14 05:05:11.345521 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:11.345498 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c366d3-zmmcw"] Sep 14 05:05:11.552628 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:05:11.552599 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28714f34-228b-400a-be2f-48015a3258c4" path="/var/lib/kubelet/pods/28714f34-228b-400a-be2f-48015a3258c4/volumes" Sep 14 05:06:05.256353 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.256281 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:05.256827 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.256737 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28714f34-228b-400a-be2f-48015a3258c4" containerName="revoke-keys" Sep 14 05:06:05.256827 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.256751 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="28714f34-228b-400a-be2f-48015a3258c4" containerName="revoke-keys" Sep 14 05:06:05.256926 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.256837 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="28714f34-228b-400a-be2f-48015a3258c4" containerName="revoke-keys" Sep 14 05:06:05.260265 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.260247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.262022 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.262003 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4ee4bdd9\"" Sep 14 05:06:05.274347 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.274325 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:05.302533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.302513 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:05.306079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.306064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.307735 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.307715 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4ee4bdd9-dockercfg-5pr4g\"" Sep 14 05:06:05.315927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.315905 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:05.345073 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.345052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlwk\" (UniqueName: \"kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.345195 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.345180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.446615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.446580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlwk\" (UniqueName: \"kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.446781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.446627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.446781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.446700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.446875 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.446779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pt4\" (UniqueName: \"kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.447359 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.447341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.453470 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.453450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlwk\" (UniqueName: \"kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk\") pod \"payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.548042 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.547979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.548166 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.548147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pt4\" (UniqueName: \"kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.548685 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.548663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.560143 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.556298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pt4\" (UniqueName: \"kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4\") pod \"payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.570624 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.570602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:05.618549 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.618523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:05.842202 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.842174 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:05.843765 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:06:05.843740 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea10c15e_ed9d_4ece_b5bb_03386a28b622.slice/crio-cbdc297d9212beef348507f370e68535648f373fb05d01b10f4c3b56949d991a WatchSource:0}: Error finding container cbdc297d9212beef348507f370e68535648f373fb05d01b10f4c3b56949d991a: Status 404 returned error can't find the container with id cbdc297d9212beef348507f370e68535648f373fb05d01b10f4c3b56949d991a Sep 14 05:06:05.913508 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:05.913485 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:05.915207 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:06:05.915180 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d10d44_1a23_47c1_a5bf_6ae0a3c9b007.slice/crio-abbc6099a970dbac42c5f9dd6ea24af383ac27f6943e7de3b4685cb4910885e9 WatchSource:0}: Error finding container abbc6099a970dbac42c5f9dd6ea24af383ac27f6943e7de3b4685cb4910885e9: Status 404 returned error can't find the container with id abbc6099a970dbac42c5f9dd6ea24af383ac27f6943e7de3b4685cb4910885e9 Sep 14 05:06:06.232002 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.231965 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" event={"ID":"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007","Type":"ContainerStarted","Data":"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f"} Sep 14 05:06:06.232203 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.232022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" event={"ID":"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007","Type":"ContainerStarted","Data":"abbc6099a970dbac42c5f9dd6ea24af383ac27f6943e7de3b4685cb4910885e9"} Sep 14 05:06:06.232203 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.232151 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:06.233756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.233732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" event={"ID":"ea10c15e-ed9d-4ece-b5bb-03386a28b622","Type":"ContainerStarted","Data":"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974"} Sep 14 05:06:06.233938 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.233775 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" event={"ID":"ea10c15e-ed9d-4ece-b5bb-03386a28b622","Type":"ContainerStarted","Data":"cbdc297d9212beef348507f370e68535648f373fb05d01b10f4c3b56949d991a"} Sep 14 05:06:06.233938 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.233823 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:06.249033 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.248981 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" podStartSLOduration=1.248965043 podStartE2EDuration="1.248965043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:06:06.247705139 +0000 UTC m=+1177.205321877" watchObservedRunningTime="2026-09-14 05:06:06.248965043 +0000 UTC m=+1177.206581781" Sep 14 05:06:06.263256 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:06.263208 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" podStartSLOduration=1.263192204 podStartE2EDuration="1.263192204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:06:06.261052547 +0000 UTC m=+1177.218669284" watchObservedRunningTime="2026-09-14 05:06:06.263192204 +0000 UTC m=+1177.220808942" Sep 14 05:06:17.241306 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:17.241277 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:17.241791 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:17.241490 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:38.059612 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.059578 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l"] Sep 14 05:06:38.063844 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.063822 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:38.065733 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.065711 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:06:38.065864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.065842 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:06:38.070261 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.070237 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l"] Sep 14 05:06:38.135642 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.135618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l\" (UID: \"ebea79ac-122f-4a27-a9a4-144f8e120a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:38.236189 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.236155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l\" (UID: \"ebea79ac-122f-4a27-a9a4-144f8e120a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:38.236723 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.236705 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l\" (UID: \"ebea79ac-122f-4a27-a9a4-144f8e120a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:38.375850 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.375827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:38.649710 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:38.649683 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l"] Sep 14 05:06:38.652009 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:06:38.651981 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebea79ac_122f_4a27_a9a4_144f8e120a0e.slice/crio-de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b WatchSource:0}: Error finding container de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b: Status 404 returned error can't find the container with id de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b Sep 14 05:06:39.414648 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:39.414615 2581 generic.go:356] "Generic (PLEG): container finished" podID="ebea79ac-122f-4a27-a9a4-144f8e120a0e" containerID="cc4490bacf807d99334f592d41250ac2b24778555e937d5979f64e56d2bf0ec3" exitCode=0 Sep 14 05:06:39.415049 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:39.414705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" event={"ID":"ebea79ac-122f-4a27-a9a4-144f8e120a0e","Type":"ContainerDied","Data":"cc4490bacf807d99334f592d41250ac2b24778555e937d5979f64e56d2bf0ec3"} Sep 14 05:06:39.415049 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:39.414811 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" event={"ID":"ebea79ac-122f-4a27-a9a4-144f8e120a0e","Type":"ContainerStarted","Data":"de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b"} Sep 14 05:06:40.572446 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:40.572419 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:40.661289 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:40.661254 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca\") pod \"ebea79ac-122f-4a27-a9a4-144f8e120a0e\" (UID: \"ebea79ac-122f-4a27-a9a4-144f8e120a0e\") " Sep 14 05:06:40.661669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:40.661647 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebea79ac-122f-4a27-a9a4-144f8e120a0e" (UID: "ebea79ac-122f-4a27-a9a4-144f8e120a0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:06:40.763253 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:40.763174 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea79ac-122f-4a27-a9a4-144f8e120a0e-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:06:41.426635 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:41.426606 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" Sep 14 05:06:41.426820 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:41.426606 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l" event={"ID":"ebea79ac-122f-4a27-a9a4-144f8e120a0e","Type":"ContainerDied","Data":"de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b"} Sep 14 05:06:41.426820 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:41.426736 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de8d932c2e28c3c2429bc79e57834fedf74b2309ed23041e10f1c9edfd77e40b" Sep 14 05:06:48.576232 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.576199 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:48.576773 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.576450 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" podUID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" containerName="payload-processing" containerID="cri-o://a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f" gracePeriod=30 Sep 14 05:06:48.583760 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.583708 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:48.583947 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.583923 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" podUID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" containerName="payload-pre-processing" containerID="cri-o://d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974" gracePeriod=30 Sep 14 05:06:48.733932 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.733877 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l"] Sep 14 05:06:48.736486 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.736460 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ee4bdd9-xqt8l"] Sep 14 05:06:48.884948 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.884923 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:48.887775 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:48.887754 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:49.049772 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.049741 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7pt4\" (UniqueName: \"kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4\") pod \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " Sep 14 05:06:49.049938 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.049790 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjlwk\" (UniqueName: \"kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk\") pod \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " Sep 14 05:06:49.049938 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.049828 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume\") pod \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\" (UID: \"ea10c15e-ed9d-4ece-b5bb-03386a28b622\") " Sep 14 05:06:49.050076 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.049984 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume\") pod \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\" (UID: \"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007\") " Sep 14 05:06:49.050261 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.050239 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea10c15e-ed9d-4ece-b5bb-03386a28b622" (UID: "ea10c15e-ed9d-4ece-b5bb-03386a28b622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:06:49.050397 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.050372 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" (UID: "69d10d44-1a23-47c1-a5bf-6ae0a3c9b007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:06:49.050469 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.050438 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:06:49.050469 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.050453 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea10c15e-ed9d-4ece-b5bb-03386a28b622-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:06:49.051941 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.051918 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4" (OuterVolumeSpecName: "kube-api-access-z7pt4") pod "69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" (UID: "69d10d44-1a23-47c1-a5bf-6ae0a3c9b007"). InnerVolumeSpecName "kube-api-access-z7pt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:06:49.052034 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.052019 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk" (OuterVolumeSpecName: "kube-api-access-cjlwk") pod "ea10c15e-ed9d-4ece-b5bb-03386a28b622" (UID: "ea10c15e-ed9d-4ece-b5bb-03386a28b622"). InnerVolumeSpecName "kube-api-access-cjlwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:06:49.151672 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.151598 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pt4\" (UniqueName: \"kubernetes.io/projected/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007-kube-api-access-z7pt4\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:06:49.151672 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.151625 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlwk\" (UniqueName: \"kubernetes.io/projected/ea10c15e-ed9d-4ece-b5bb-03386a28b622-kube-api-access-cjlwk\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:06:49.472111 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.472017 2581 generic.go:356] "Generic (PLEG): container finished" podID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" containerID="a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f" exitCode=0 Sep 14 05:06:49.472111 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.472066 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" Sep 14 05:06:49.472111 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.472083 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" event={"ID":"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007","Type":"ContainerDied","Data":"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f"} Sep 14 05:06:49.472631 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.472189 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx" event={"ID":"69d10d44-1a23-47c1-a5bf-6ae0a3c9b007","Type":"ContainerDied","Data":"abbc6099a970dbac42c5f9dd6ea24af383ac27f6943e7de3b4685cb4910885e9"} Sep 14 05:06:49.472631 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.472219 2581 scope.go:117] "RemoveContainer" containerID="a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f" Sep 14 05:06:49.473895 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.473868 2581 generic.go:356] "Generic (PLEG): container finished" podID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" containerID="d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974" exitCode=0 Sep 14 05:06:49.474013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.473938 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" Sep 14 05:06:49.474013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.473947 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" event={"ID":"ea10c15e-ed9d-4ece-b5bb-03386a28b622","Type":"ContainerDied","Data":"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974"} Sep 14 05:06:49.474013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.474005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm" event={"ID":"ea10c15e-ed9d-4ece-b5bb-03386a28b622","Type":"ContainerDied","Data":"cbdc297d9212beef348507f370e68535648f373fb05d01b10f4c3b56949d991a"} Sep 14 05:06:49.482126 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.482096 2581 scope.go:117] "RemoveContainer" containerID="a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f" Sep 14 05:06:49.482876 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:06:49.482595 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f\": container with ID starting with a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f not found: ID does not exist" containerID="a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f" Sep 14 05:06:49.482876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.482631 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f"} err="failed to get container status \"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f\": rpc error: code = NotFound desc = could not find container \"a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f\": container with ID starting with a002cd489d01808d55415a85baab1861e758db85700a68c0ecb70b67ea4d656f not found: ID does not exist" Sep 14 05:06:49.482876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.482651 2581 scope.go:117] "RemoveContainer" containerID="d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974" Sep 14 05:06:49.492381 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.492361 2581 scope.go:117] "RemoveContainer" containerID="d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974" Sep 14 05:06:49.492623 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:06:49.492605 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974\": container with ID starting with d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974 not found: ID does not exist" containerID="d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974" Sep 14 05:06:49.492698 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.492640 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974"} err="failed to get container status \"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974\": rpc error: code = NotFound desc = could not find container \"d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974\": container with ID starting with d99a0e882a50e8cbdc9156d87b58650d4b53b7eee3b6d9135d4db53e84900974 not found: ID does not exist" Sep 14 05:06:49.500418 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.500394 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:49.504493 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.504466 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ee4bdd9-69fbb96966-x8jdm"] Sep 14 05:06:49.511632 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.511607 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:49.516727 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.516702 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ee4bdd9-5db967f57c-ggtgx"] Sep 14 05:06:49.553523 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.553487 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" path="/var/lib/kubelet/pods/69d10d44-1a23-47c1-a5bf-6ae0a3c9b007/volumes" Sep 14 05:06:49.553863 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.553848 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" path="/var/lib/kubelet/pods/ea10c15e-ed9d-4ece-b5bb-03386a28b622/volumes" Sep 14 05:06:49.554205 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:06:49.554188 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebea79ac-122f-4a27-a9a4-144f8e120a0e" path="/var/lib/kubelet/pods/ebea79ac-122f-4a27-a9a4-144f8e120a0e/volumes" Sep 14 05:07:09.730924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.730891 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:07:09.731379 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731364 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebea79ac-122f-4a27-a9a4-144f8e120a0e" containerName="revoke-keys" Sep 14 05:07:09.731379 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731380 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebea79ac-122f-4a27-a9a4-144f8e120a0e" containerName="revoke-keys" Sep 14 05:07:09.731473 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731400 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" containerName="payload-pre-processing" Sep 14 05:07:09.731473 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731405 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" containerName="payload-pre-processing" Sep 14 05:07:09.731473 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731418 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" containerName="payload-processing" Sep 14 05:07:09.731473 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731424 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" containerName="payload-processing" Sep 14 05:07:09.731613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731483 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea10c15e-ed9d-4ece-b5bb-03386a28b622" containerName="payload-pre-processing" Sep 14 05:07:09.731613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731489 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebea79ac-122f-4a27-a9a4-144f8e120a0e" containerName="revoke-keys" Sep 14 05:07:09.731613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.731500 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d10d44-1a23-47c1-a5bf-6ae0a3c9b007" containerName="payload-processing" Sep 14 05:07:09.734926 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.734905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.736864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.736842 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2eeb50-openshift-default-dockercfg-g6bhj\"" Sep 14 05:07:09.746055 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.746032 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:07:09.857164 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.857127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.857777 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.857748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpktc\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858037 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858336 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858444 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858398 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858508 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858566 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858566 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.858671 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.858607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959397 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959397 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959616 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959691 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959746 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959800 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959742 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959863 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959916 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.959958 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.960000 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpktc\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.960000 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.959983 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.960080 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.960018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.960347 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.960320 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.960631 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.960614 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.961804 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.961783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.962227 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.962197 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.974721 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.974695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpktc\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:09.975095 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:09.975079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token\") pod \"e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:10.048382 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.048290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:10.326478 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.326291 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:07:10.329365 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:07:10.329336 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48713d24_602b_4485_9f76_5466dba96659.slice/crio-cac6ee92f636bfba59f26a8b9782693097d4eaf6598536893ced49f0d3b473c9 WatchSource:0}: Error finding container cac6ee92f636bfba59f26a8b9782693097d4eaf6598536893ced49f0d3b473c9: Status 404 returned error can't find the container with id cac6ee92f636bfba59f26a8b9782693097d4eaf6598536893ced49f0d3b473c9 Sep 14 05:07:10.331566 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.331530 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:10.331694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.331608 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:10.331694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.331654 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:10.594368 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.594268 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerStarted","Data":"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b"} Sep 14 05:07:10.594368 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.594338 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerStarted","Data":"cac6ee92f636bfba59f26a8b9782693097d4eaf6598536893ced49f0d3b473c9"} Sep 14 05:07:10.615009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:10.614951 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podStartSLOduration=1.614935763 podStartE2EDuration="1.614935763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:10.61187628 +0000 UTC m=+1241.569493029" watchObservedRunningTime="2026-09-14 05:07:10.614935763 +0000 UTC m=+1241.572552499" Sep 14 05:07:11.048856 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:11.048825 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:11.053986 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:11.053959 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:11.600192 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:11.600155 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:11.601606 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:11.601585 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:16.753465 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.753429 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:07:16.757506 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.757487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.759340 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.759318 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2eeb50-dockercfg-2wv7k\"" Sep 14 05:07:16.759443 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.759390 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2eeb50\"" Sep 14 05:07:16.768219 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.768191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:07:16.834091 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.833773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.834091 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.833937 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8grj\" (UniqueName: \"kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.935448 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.935415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8grj\" (UniqueName: \"kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.935627 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.935502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.936209 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.936190 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:16.942030 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:16.942004 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8grj\" (UniqueName: \"kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj\") pod \"payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:17.070602 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.070539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:17.344309 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.344285 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:07:17.346269 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:07:17.346241 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0cc9c6_00d4_4b17_b212_5e8aac5eda36.slice/crio-9edf2041a4bfaabfae1266bdfa41681ef40134dccd1108f9dd3644c5183e5787 WatchSource:0}: Error finding container 9edf2041a4bfaabfae1266bdfa41681ef40134dccd1108f9dd3644c5183e5787: Status 404 returned error can't find the container with id 9edf2041a4bfaabfae1266bdfa41681ef40134dccd1108f9dd3644c5183e5787 Sep 14 05:07:17.637223 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.637188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" event={"ID":"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36","Type":"ContainerStarted","Data":"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663"} Sep 14 05:07:17.637390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.637237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" event={"ID":"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36","Type":"ContainerStarted","Data":"9edf2041a4bfaabfae1266bdfa41681ef40134dccd1108f9dd3644c5183e5787"} Sep 14 05:07:17.637390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.637327 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:17.653087 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:17.653041 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" podStartSLOduration=1.653022545 podStartE2EDuration="1.653022545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:17.65174575 +0000 UTC m=+1248.609362488" watchObservedRunningTime="2026-09-14 05:07:17.653022545 +0000 UTC m=+1248.610639283" Sep 14 05:07:28.646401 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:28.646304 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:07:36.749928 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.749845 2581 generic.go:356] "Generic (PLEG): container finished" podID="48713d24-602b-4485-9f76-5466dba96659" containerID="92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b" exitCode=0 Sep 14 05:07:36.750445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.749923 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerDied","Data":"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b"} Sep 14 05:07:36.750506 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.750489 2581 scope.go:117] "RemoveContainer" containerID="92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b" Sep 14 05:07:36.751621 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.751593 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:36.751732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.751675 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:36.751732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:36.751719 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:37.762664 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:37.762623 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerStarted","Data":"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b"} Sep 14 05:07:38.050338 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:38.050245 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:38.050513 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:38.050425 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 05:07:38.050513 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:38.050475 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 05:07:39.049536 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:39.049497 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 05:07:39.050133 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:39.050086 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 05:07:40.050491 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:40.050448 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 05:07:40.050987 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:40.050527 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 05:07:41.049049 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:41.049007 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 05:07:41.049267 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:41.049073 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 05:07:41.600822 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:41.600791 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:42.049070 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:42.049034 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 14 05:07:42.049376 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:42.049348 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 14 05:07:43.142616 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.142581 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:43.144931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.144905 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:07:43.618568 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.616651 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:07:43.623260 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.623214 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.625102 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.625078 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a8c46f-openshift-default-dockercfg-s8z64\"" Sep 14 05:07:43.625785 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.625762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:07:43.722437 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.722399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.722714 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.722688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtw2r\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.722840 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.722825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.722956 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.722940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.723057 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.723043 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.723189 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.723172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.723372 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.723354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.723501 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.723471 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.723641 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.723589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825147 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtw2r\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825147 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825703 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825703 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825589 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825703 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825609 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825844 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.825909 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.825863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.826065 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.826034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.827693 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.827673 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.827990 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.827971 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.831661 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.831639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.831940 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.831923 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtw2r\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r\") pod \"e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:43.935597 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:43.935530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:44.212489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.212464 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:07:44.214533 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:07:44.214508 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a115b4_27de_4cf3_8ed7_3b9383d03463.slice/crio-6defe7ef19566d74ace45036f25854fd48dde74e3b69f0b0fed8e9331c6bd708 WatchSource:0}: Error finding container 6defe7ef19566d74ace45036f25854fd48dde74e3b69f0b0fed8e9331c6bd708: Status 404 returned error can't find the container with id 6defe7ef19566d74ace45036f25854fd48dde74e3b69f0b0fed8e9331c6bd708 Sep 14 05:07:44.217348 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.217317 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:44.217434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.217388 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:44.217434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.217418 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:07:44.813487 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.813455 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerStarted","Data":"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa"} Sep 14 05:07:44.813651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.813502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerStarted","Data":"6defe7ef19566d74ace45036f25854fd48dde74e3b69f0b0fed8e9331c6bd708"} Sep 14 05:07:44.833283 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.833237 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podStartSLOduration=1.833219262 podStartE2EDuration="1.833219262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:44.831583996 +0000 UTC m=+1275.789200733" watchObservedRunningTime="2026-09-14 05:07:44.833219262 +0000 UTC m=+1275.790836000" Sep 14 05:07:44.935824 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:44.935789 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:45.940919 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:45.940891 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:46.826031 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:46.826004 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:07:46.827041 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:07:46.827022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:08:13.845953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.845916 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2"] Sep 14 05:08:13.856581 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.856554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:13.858749 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.858728 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:08:13.858918 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.858897 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2"] Sep 14 05:08:13.859059 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.859039 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:08:13.969806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:13.969769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2\" (UID: \"6b18b004-8936-494f-b0bc-6e7f433169da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:14.070489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.070459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2\" (UID: \"6b18b004-8936-494f-b0bc-6e7f433169da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:14.071038 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.071017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2\" (UID: \"6b18b004-8936-494f-b0bc-6e7f433169da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:14.172473 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.172443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:14.444902 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.444865 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2"] Sep 14 05:08:14.458138 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:08:14.457271 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b18b004_8936_494f_b0bc_6e7f433169da.slice/crio-4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6 WatchSource:0}: Error finding container 4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6: Status 404 returned error can't find the container with id 4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6 Sep 14 05:08:14.995060 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.995023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" event={"ID":"6b18b004-8936-494f-b0bc-6e7f433169da","Type":"ContainerStarted","Data":"421fc825a39dcac79a1c2f4b7d663a68f769f11298d5463bcdcf2b611e2c1c4f"} Sep 14 05:08:14.995439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:14.995081 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" event={"ID":"6b18b004-8936-494f-b0bc-6e7f433169da","Type":"ContainerStarted","Data":"4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6"} Sep 14 05:08:15.008176 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:15.007660 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" podStartSLOduration=2.007641069 podStartE2EDuration="2.007641069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:15.007111233 +0000 UTC m=+1305.964727969" watchObservedRunningTime="2026-09-14 05:08:15.007641069 +0000 UTC m=+1305.965257810" Sep 14 05:08:16.003598 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:16.003562 2581 generic.go:356] "Generic (PLEG): container finished" podID="6b18b004-8936-494f-b0bc-6e7f433169da" containerID="421fc825a39dcac79a1c2f4b7d663a68f769f11298d5463bcdcf2b611e2c1c4f" exitCode=0 Sep 14 05:08:16.004004 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:16.003617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" event={"ID":"6b18b004-8936-494f-b0bc-6e7f433169da","Type":"ContainerDied","Data":"421fc825a39dcac79a1c2f4b7d663a68f769f11298d5463bcdcf2b611e2c1c4f"} Sep 14 05:08:17.188722 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.188694 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:17.305419 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.305336 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca\") pod \"6b18b004-8936-494f-b0bc-6e7f433169da\" (UID: \"6b18b004-8936-494f-b0bc-6e7f433169da\") " Sep 14 05:08:17.305810 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.305780 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b18b004-8936-494f-b0bc-6e7f433169da" (UID: "6b18b004-8936-494f-b0bc-6e7f433169da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:08:17.406884 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.406847 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b18b004-8936-494f-b0bc-6e7f433169da-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:08:17.867372 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.867333 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 05:08:17.867713 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:17.867421 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 05:08:18.018375 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:18.018338 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" Sep 14 05:08:18.018375 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:18.018361 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2" event={"ID":"6b18b004-8936-494f-b0bc-6e7f433169da","Type":"ContainerDied","Data":"4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6"} Sep 14 05:08:18.018595 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:18.018423 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef50b67902fbf5567ceb53d2d6ab6aca6fafc556b67221bcac982bc35c513a6" Sep 14 05:08:19.025050 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.025018 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerID="520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa" exitCode=0 Sep 14 05:08:19.025593 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.025067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerDied","Data":"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa"} Sep 14 05:08:19.025593 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.025567 2581 scope.go:117] "RemoveContainer" containerID="520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa" Sep 14 05:08:19.026816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.026787 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:08:19.026889 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.026857 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:08:19.026927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:19.026890 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:08:20.033798 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:20.033762 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerStarted","Data":"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e"} Sep 14 05:08:20.935967 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:20.935935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:08:20.936149 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:20.936105 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 05:08:20.936225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:20.936175 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 05:08:21.936857 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:21.936815 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 05:08:21.937354 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:21.936885 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 05:08:22.936078 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:22.936039 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 05:08:22.936308 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:22.936097 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 05:08:23.936391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:23.936355 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 05:08:23.936966 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:23.936929 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 05:08:24.761953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:24.761913 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2"] Sep 14 05:08:24.765690 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:24.765662 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9529a1-l9qh2"] Sep 14 05:08:24.936078 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:24.936041 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 14 05:08:24.936308 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:24.936105 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 14 05:08:25.553822 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:25.553794 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b18b004-8936-494f-b0bc-6e7f433169da" path="/var/lib/kubelet/pods/6b18b004-8936-494f-b0bc-6e7f433169da/volumes" Sep 14 05:08:25.940251 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:25.940215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:08:25.940507 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:25.940486 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:08:25.941227 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:25.941211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:08:37.627716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.627684 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:08:37.629140 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.629088 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b18b004-8936-494f-b0bc-6e7f433169da" containerName="revoke-keys" Sep 14 05:08:37.629302 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.629288 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b18b004-8936-494f-b0bc-6e7f433169da" containerName="revoke-keys" Sep 14 05:08:37.629538 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.629519 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b18b004-8936-494f-b0bc-6e7f433169da" containerName="revoke-keys" Sep 14 05:08:37.633553 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.633530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.638782 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.637709 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7c6a48c\"" Sep 14 05:08:37.638782 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.638557 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7c6a48c-dockercfg-2lgk2\"" Sep 14 05:08:37.640645 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.640614 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:08:37.714600 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.714576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lz88\" (UniqueName: \"kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.714745 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.714636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.816310 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.816277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lz88\" (UniqueName: \"kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.816433 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.816342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.817019 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.816998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.823207 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.823190 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lz88\" (UniqueName: \"kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88\") pod \"payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:37.950839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:37.950778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:38.228498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:38.221571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:08:39.168018 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:39.167977 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" event={"ID":"93bb0224-4827-4e75-95b2-90c358d40f72","Type":"ContainerStarted","Data":"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72"} Sep 14 05:08:39.168422 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:39.168034 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" event={"ID":"93bb0224-4827-4e75-95b2-90c358d40f72","Type":"ContainerStarted","Data":"6bc8c9ac33af5bac117d728cbd36b9547a4c6121cba65997a4ce6ec2770d6fe2"} Sep 14 05:08:39.168422 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:39.168073 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:08:39.185886 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:39.185845 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" podStartSLOduration=2.185831977 podStartE2EDuration="2.185831977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:39.183096044 +0000 UTC m=+1330.140712781" watchObservedRunningTime="2026-09-14 05:08:39.185831977 +0000 UTC m=+1330.143448714" Sep 14 05:08:50.175498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:08:50.175469 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:09:29.648951 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:29.648925 2581 scope.go:117] "RemoveContainer" containerID="b4385e7e6265c4023310d24ba0d3ba2156c1dcea16a9770229e95d1a330c5ece" Sep 14 05:09:45.311352 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.311309 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59"] Sep 14 05:09:45.316653 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.316634 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:45.318448 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.318427 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:09:45.318873 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.318851 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:09:45.320484 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.320458 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59"] Sep 14 05:09:45.359706 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.359677 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-667664-j4f59\" (UID: \"d50d0a73-0012-4309-929a-5867fe4afbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:45.460656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.460617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-667664-j4f59\" (UID: \"d50d0a73-0012-4309-929a-5867fe4afbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:45.461331 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.461314 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-667664-j4f59\" (UID: \"d50d0a73-0012-4309-929a-5867fe4afbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:45.629599 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.629563 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:45.898535 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:45.898511 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59"] Sep 14 05:09:45.900807 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:09:45.900779 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50d0a73_0012_4309_929a_5867fe4afbc9.slice/crio-8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0 WatchSource:0}: Error finding container 8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0: Status 404 returned error can't find the container with id 8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0 Sep 14 05:09:46.565285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:46.565258 2581 generic.go:356] "Generic (PLEG): container finished" podID="d50d0a73-0012-4309-929a-5867fe4afbc9" containerID="b248c45d3f1054f86bf00f06b510e43b56d2788cca354a5bc5862a966d66cb22" exitCode=0 Sep 14 05:09:46.565632 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:46.565337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" event={"ID":"d50d0a73-0012-4309-929a-5867fe4afbc9","Type":"ContainerDied","Data":"b248c45d3f1054f86bf00f06b510e43b56d2788cca354a5bc5862a966d66cb22"} Sep 14 05:09:46.565632 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:46.565383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" event={"ID":"d50d0a73-0012-4309-929a-5867fe4afbc9","Type":"ContainerStarted","Data":"8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0"} Sep 14 05:09:47.719646 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:47.719622 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:47.781399 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:47.781362 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca\") pod \"d50d0a73-0012-4309-929a-5867fe4afbc9\" (UID: \"d50d0a73-0012-4309-929a-5867fe4afbc9\") " Sep 14 05:09:47.781804 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:47.781777 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d50d0a73-0012-4309-929a-5867fe4afbc9" (UID: "d50d0a73-0012-4309-929a-5867fe4afbc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:09:47.882520 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:47.882488 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50d0a73-0012-4309-929a-5867fe4afbc9-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:09:48.579219 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:48.579185 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" Sep 14 05:09:48.579219 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:48.579209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59" event={"ID":"d50d0a73-0012-4309-929a-5867fe4afbc9","Type":"ContainerDied","Data":"8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0"} Sep 14 05:09:48.579434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:48.579269 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db13033a7e74b8b817d85552a09ffd90a9f6799af8c33f935b40c9b64b354f0" Sep 14 05:09:55.816654 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:55.816621 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:09:55.817615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:55.817562 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" podUID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" containerName="payload-processing" containerID="cri-o://d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc" gracePeriod=30 Sep 14 05:09:55.968864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:55.968831 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59"] Sep 14 05:09:55.971398 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:55.971371 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-667664-j4f59"] Sep 14 05:09:56.123955 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.123932 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:09:56.174516 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.174475 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume\") pod \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " Sep 14 05:09:56.174719 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.174677 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bgx\" (UniqueName: \"kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx\") pod \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\" (UID: \"6db71e67-e9a1-4665-bb0a-3df67d5f3423\") " Sep 14 05:09:56.174950 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.174920 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6db71e67-e9a1-4665-bb0a-3df67d5f3423" (UID: "6db71e67-e9a1-4665-bb0a-3df67d5f3423"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:09:56.175172 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.175154 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db71e67-e9a1-4665-bb0a-3df67d5f3423-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:09:56.176825 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.176797 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx" (OuterVolumeSpecName: "kube-api-access-72bgx") pod "6db71e67-e9a1-4665-bb0a-3df67d5f3423" (UID: "6db71e67-e9a1-4665-bb0a-3df67d5f3423"). InnerVolumeSpecName "kube-api-access-72bgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:09:56.276568 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.276534 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bgx\" (UniqueName: \"kubernetes.io/projected/6db71e67-e9a1-4665-bb0a-3df67d5f3423-kube-api-access-72bgx\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:09:56.626840 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.626803 2581 generic.go:356] "Generic (PLEG): container finished" podID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" containerID="d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc" exitCode=0 Sep 14 05:09:56.627009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.626857 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" Sep 14 05:09:56.627009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.626890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" event={"ID":"6db71e67-e9a1-4665-bb0a-3df67d5f3423","Type":"ContainerDied","Data":"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc"} Sep 14 05:09:56.627009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.626937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r" event={"ID":"6db71e67-e9a1-4665-bb0a-3df67d5f3423","Type":"ContainerDied","Data":"bd04ff6ad513c14ee1f36f569718b6f883819de30dd9e172f46e7f532a9a13fd"} Sep 14 05:09:56.627009 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.626960 2581 scope.go:117] "RemoveContainer" containerID="d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc" Sep 14 05:09:56.636101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.636082 2581 scope.go:117] "RemoveContainer" containerID="d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc" Sep 14 05:09:56.636438 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:09:56.636419 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc\": container with ID starting with d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc not found: ID does not exist" containerID="d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc" Sep 14 05:09:56.636507 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.636450 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc"} err="failed to get container status \"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc\": rpc error: code = NotFound desc = could not find container \"d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc\": container with ID starting with d260157a50a42339c9cd555853d666f90ae9cf2c078bfc50e43ffbcad0bf60cc not found: ID does not exist" Sep 14 05:09:56.647609 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.647559 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:09:56.650360 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:56.650339 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-667664-5bdbdc7d59-lck8r"] Sep 14 05:09:57.553319 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:57.553289 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" path="/var/lib/kubelet/pods/6db71e67-e9a1-4665-bb0a-3df67d5f3423/volumes" Sep 14 05:09:57.553676 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:09:57.553620 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50d0a73-0012-4309-929a-5867fe4afbc9" path="/var/lib/kubelet/pods/d50d0a73-0012-4309-929a-5867fe4afbc9/volumes" Sep 14 05:10:02.672516 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:02.672480 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:10:02.672907 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:02.672765 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" containerID="cri-o://7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5" gracePeriod=30 Sep 14 05:10:07.920691 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.920666 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:10:07.995482 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995390 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995482 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995440 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jj9d\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995503 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995648 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995720 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995765 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995801 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.995839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995825 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket\") pod \"61027dab-03ad-42a6-be36-5ee55a34c36b\" (UID: \"61027dab-03ad-42a6-be36-5ee55a34c36b\") " Sep 14 05:10:07.996097 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.995835 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:10:07.996097 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996029 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:07.996231 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996162 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:07.996280 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996248 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data" (OuterVolumeSpecName: "istio-data") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:07.996334 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996317 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:07.996388 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996338 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:07.996388 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996353 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61027dab-03ad-42a6-be36-5ee55a34c36b-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:07.996476 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.996407 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:07.997871 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.997839 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:10:07.997977 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.997887 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:07.998085 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.998062 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token" (OuterVolumeSpecName: "istio-token") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:07.998143 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:07.998094 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d" (OuterVolumeSpecName: "kube-api-access-9jj9d") pod "61027dab-03ad-42a6-be36-5ee55a34c36b" (UID: "61027dab-03ad-42a6-be36-5ee55a34c36b"). InnerVolumeSpecName "kube-api-access-9jj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:08.097367 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097333 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.097367 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097360 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.097367 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097369 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.097367 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097377 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.097635 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097387 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jj9d\" (UniqueName: \"kubernetes.io/projected/61027dab-03ad-42a6-be36-5ee55a34c36b-kube-api-access-9jj9d\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.097635 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.097395 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61027dab-03ad-42a6-be36-5ee55a34c36b-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:08.699645 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.699616 2581 generic.go:356] "Generic (PLEG): container finished" podID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerID="7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5" exitCode=0 Sep 14 05:10:08.699799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.699665 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" Sep 14 05:10:08.699799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.699689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerDied","Data":"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5"} Sep 14 05:10:08.699799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.699742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n" event={"ID":"61027dab-03ad-42a6-be36-5ee55a34c36b","Type":"ContainerDied","Data":"cdddc3ef58fec2b655568957280c4112260a6a156364917ac2915811dfb83e73"} Sep 14 05:10:08.699799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.699764 2581 scope.go:117] "RemoveContainer" containerID="7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5" Sep 14 05:10:08.713282 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.713254 2581 scope.go:117] "RemoveContainer" containerID="fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468" Sep 14 05:10:08.722447 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.722414 2581 scope.go:117] "RemoveContainer" containerID="7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5" Sep 14 05:10:08.722809 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:08.722781 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5\": container with ID starting with 7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5 not found: ID does not exist" containerID="7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5" Sep 14 05:10:08.722859 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.722827 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5"} err="failed to get container status \"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5\": rpc error: code = NotFound desc = could not find container \"7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5\": container with ID starting with 7acd15badd5abbbacb247faa55dbd200f4b4e4e8c7797829bfee4dbeac57ccc5 not found: ID does not exist" Sep 14 05:10:08.722859 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.722849 2581 scope.go:117] "RemoveContainer" containerID="fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468" Sep 14 05:10:08.723194 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:08.723163 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468\": container with ID starting with fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468 not found: ID does not exist" containerID="fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468" Sep 14 05:10:08.723246 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.723202 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468"} err="failed to get container status \"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468\": rpc error: code = NotFound desc = could not find container \"fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468\": container with ID starting with fc4d4192ff911e7cdda5038931b69bb45a9662128aae5599040a8b673b5de468 not found: ID does not exist" Sep 14 05:10:08.726308 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.726284 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:10:08.729535 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:08.729511 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-667664-openshift-default-8675dc4d6d-g7d2n"] Sep 14 05:10:09.553273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:09.553236 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" path="/var/lib/kubelet/pods/61027dab-03ad-42a6-be36-5ee55a34c36b/volumes" Sep 14 05:10:13.637244 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:13.637206 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:10:13.638067 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:13.638035 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" podUID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" containerName="payload-pre-processing" containerID="cri-o://dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299" gracePeriod=30 Sep 14 05:10:13.940109 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:13.940076 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:10:14.060177 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.060148 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume\") pod \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " Sep 14 05:10:14.060362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.060328 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwqrl\" (UniqueName: \"kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl\") pod \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\" (UID: \"41de4b1f-5003-4dd7-a72f-e52df4c2ef43\") " Sep 14 05:10:14.060508 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.060480 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41de4b1f-5003-4dd7-a72f-e52df4c2ef43" (UID: "41de4b1f-5003-4dd7-a72f-e52df4c2ef43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:10:14.060672 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.060655 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:14.062422 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.062398 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl" (OuterVolumeSpecName: "kube-api-access-pwqrl") pod "41de4b1f-5003-4dd7-a72f-e52df4c2ef43" (UID: "41de4b1f-5003-4dd7-a72f-e52df4c2ef43"). InnerVolumeSpecName "kube-api-access-pwqrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:14.162169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.162082 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqrl\" (UniqueName: \"kubernetes.io/projected/41de4b1f-5003-4dd7-a72f-e52df4c2ef43-kube-api-access-pwqrl\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:14.736467 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.736432 2581 generic.go:356] "Generic (PLEG): container finished" podID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" containerID="dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299" exitCode=0 Sep 14 05:10:14.736865 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.736475 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" Sep 14 05:10:14.736865 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.736509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" event={"ID":"41de4b1f-5003-4dd7-a72f-e52df4c2ef43","Type":"ContainerDied","Data":"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299"} Sep 14 05:10:14.736865 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.736559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf" event={"ID":"41de4b1f-5003-4dd7-a72f-e52df4c2ef43","Type":"ContainerDied","Data":"0b664c648e80562d30c1e4d019ca2a6160e6c5424e7f9e6a3d3d3253ac03f835"} Sep 14 05:10:14.736865 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.736582 2581 scope.go:117] "RemoveContainer" containerID="dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299" Sep 14 05:10:14.745797 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.745766 2581 scope.go:117] "RemoveContainer" containerID="dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299" Sep 14 05:10:14.746069 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:14.746051 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299\": container with ID starting with dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299 not found: ID does not exist" containerID="dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299" Sep 14 05:10:14.746129 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.746077 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299"} err="failed to get container status \"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299\": rpc error: code = NotFound desc = could not find container \"dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299\": container with ID starting with dcb6d2d1948194c4ed1eb2370a836c2a4bac182fd54bf323d76eeb8e9cb5d299 not found: ID does not exist" Sep 14 05:10:14.755939 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.755915 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:10:14.759626 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:14.759602 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7fdd6-6d5f6c59c8-kwsxf"] Sep 14 05:10:15.553495 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:15.553462 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" path="/var/lib/kubelet/pods/41de4b1f-5003-4dd7-a72f-e52df4c2ef43/volumes" Sep 14 05:10:20.504101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:20.504060 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:10:20.504612 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:20.504375 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" containerID="cri-o://fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed" gracePeriod=30 Sep 14 05:10:25.754257 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.754229 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:10:25.781330 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781271 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781330 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781316 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781442 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781474 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdckh\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781500 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781544 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:25.781737 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781579 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781737 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781611 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781737 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781652 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token\") pod \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\" (UID: \"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0\") " Sep 14 05:10:25.781737 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781694 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data" (OuterVolumeSpecName: "istio-data") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:25.781737 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781710 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:10:25.781995 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781862 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:25.781995 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.781899 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:25.782110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.782085 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.782110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.782104 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.782110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.782133 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.782110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.782147 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.782390 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.782159 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.784014 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.783973 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:10:25.784168 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.784002 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh" (OuterVolumeSpecName: "kube-api-access-pdckh") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "kube-api-access-pdckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:25.784168 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.784037 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:10:25.784262 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.784210 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token" (OuterVolumeSpecName: "istio-token") pod "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" (UID: "50ae07d0-0ec0-4d4b-9217-38cbc410fbb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:25.805998 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.805970 2581 generic.go:356] "Generic (PLEG): container finished" podID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerID="fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed" exitCode=0 Sep 14 05:10:25.806079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.806015 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" Sep 14 05:10:25.806079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.806049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerDied","Data":"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed"} Sep 14 05:10:25.806208 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.806106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698" event={"ID":"50ae07d0-0ec0-4d4b-9217-38cbc410fbb0","Type":"ContainerDied","Data":"a7f2b2fc8433f0e531e71e7567dbeedb713795e7991e82c2b7ff83084ef3d1a9"} Sep 14 05:10:25.806208 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.806139 2581 scope.go:117] "RemoveContainer" containerID="fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed" Sep 14 05:10:25.818361 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.818346 2581 scope.go:117] "RemoveContainer" containerID="b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b" Sep 14 05:10:25.827052 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.827030 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:10:25.827340 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.827318 2581 scope.go:117] "RemoveContainer" containerID="fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed" Sep 14 05:10:25.829214 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:25.829178 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed\": container with ID starting with fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed not found: ID does not exist" containerID="fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed" Sep 14 05:10:25.829292 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.829215 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed"} err="failed to get container status \"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed\": rpc error: code = NotFound desc = could not find container \"fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed\": container with ID starting with fdc97a13292f1c9fb9a1493e165ae76a4628dd43a505696122d73515585839ed not found: ID does not exist" Sep 14 05:10:25.829292 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.829237 2581 scope.go:117] "RemoveContainer" containerID="b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b" Sep 14 05:10:25.829532 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:25.829517 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b\": container with ID starting with b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b not found: ID does not exist" containerID="b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b" Sep 14 05:10:25.829583 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.829538 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b"} err="failed to get container status \"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b\": rpc error: code = NotFound desc = could not find container \"b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b\": container with ID starting with b52b7b28ea4db726c39c504025548d751c7268a4d2930c3c367fd901c855813b not found: ID does not exist" Sep 14 05:10:25.830408 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.830389 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e7fdd6-openshift-default-74bfd88f5b-64698"] Sep 14 05:10:25.883392 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.883369 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdckh\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-kube-api-access-pdckh\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.883392 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.883393 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.883511 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.883402 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:25.883511 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:25.883410 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:27.553347 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:27.553316 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" path="/var/lib/kubelet/pods/50ae07d0-0ec0-4d4b-9217-38cbc410fbb0/volumes" Sep 14 05:10:28.034258 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034227 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:10:28.034707 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034691 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50d0a73-0012-4309-929a-5867fe4afbc9" containerName="revoke-keys" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034710 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50d0a73-0012-4309-929a-5867fe4afbc9" containerName="revoke-keys" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034722 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034727 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034736 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" containerName="payload-pre-processing" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034742 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" containerName="payload-pre-processing" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034752 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" containerName="payload-processing" Sep 14 05:10:28.034758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034758 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" containerName="payload-processing" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034766 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034772 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034781 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034790 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034805 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034810 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034892 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41de4b1f-5003-4dd7-a72f-e52df4c2ef43" containerName="payload-pre-processing" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034899 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034906 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db71e67-e9a1-4665-bb0a-3df67d5f3423" containerName="payload-processing" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034913 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034920 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50d0a73-0012-4309-929a-5867fe4afbc9" containerName="revoke-keys" Sep 14 05:10:28.034985 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.034926 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61027dab-03ad-42a6-be36-5ee55a34c36b" containerName="istio-proxy" Sep 14 05:10:28.039616 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.039599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.041180 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.041156 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2ff4a6\"" Sep 14 05:10:28.041264 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.041248 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2ff4a6-dockercfg-84mbf\"" Sep 14 05:10:28.045076 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.045049 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:10:28.081139 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.081100 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:10:28.081601 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.081588 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ae07d0-0ec0-4d4b-9217-38cbc410fbb0" containerName="istio-proxy" Sep 14 05:10:28.084592 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.084577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.092786 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.092762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:10:28.104196 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.104172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.104289 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.104227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.104378 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.104356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgwq\" (UniqueName: \"kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.104421 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.104402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6df2\" (UniqueName: \"kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.205308 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.205260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgwq\" (UniqueName: \"kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.205483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.205334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6df2\" (UniqueName: \"kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.205483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.205389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.205483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.205450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.206266 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.206238 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.206418 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.206334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.212305 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.212277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6df2\" (UniqueName: \"kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2\") pod \"payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.212401 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.212379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgwq\" (UniqueName: \"kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq\") pod \"payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.351406 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.351382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.396376 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.396348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.625249 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.625222 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:10:28.626921 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:10:28.626901 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fd60b5_b337_40e4_83c7_d0beb1227996.slice/crio-b4ddd176aa7e4771505459bc7f3b1367e45448d3eb9c7690ceb4324c0c5e3eff WatchSource:0}: Error finding container b4ddd176aa7e4771505459bc7f3b1367e45448d3eb9c7690ceb4324c0c5e3eff: Status 404 returned error can't find the container with id b4ddd176aa7e4771505459bc7f3b1367e45448d3eb9c7690ceb4324c0c5e3eff Sep 14 05:10:28.682969 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.682948 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:10:28.684698 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:10:28.684672 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d78c1f_56bc_4051_8783_45f143f48749.slice/crio-2429f7fe10a7bd8a176a49de48c1519b6858112b16f8b5b51855edeb807985c0 WatchSource:0}: Error finding container 2429f7fe10a7bd8a176a49de48c1519b6858112b16f8b5b51855edeb807985c0: Status 404 returned error can't find the container with id 2429f7fe10a7bd8a176a49de48c1519b6858112b16f8b5b51855edeb807985c0 Sep 14 05:10:28.825576 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.825544 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" event={"ID":"80fd60b5-b337-40e4-83c7-d0beb1227996","Type":"ContainerStarted","Data":"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8"} Sep 14 05:10:28.825752 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.825605 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" event={"ID":"80fd60b5-b337-40e4-83c7-d0beb1227996","Type":"ContainerStarted","Data":"b4ddd176aa7e4771505459bc7f3b1367e45448d3eb9c7690ceb4324c0c5e3eff"} Sep 14 05:10:28.825752 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.825636 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:28.827202 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.827179 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" event={"ID":"a6d78c1f-56bc-4051-8783-45f143f48749","Type":"ContainerStarted","Data":"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc"} Sep 14 05:10:28.827301 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.827216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" event={"ID":"a6d78c1f-56bc-4051-8783-45f143f48749","Type":"ContainerStarted","Data":"2429f7fe10a7bd8a176a49de48c1519b6858112b16f8b5b51855edeb807985c0"} Sep 14 05:10:28.827360 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.827319 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:28.841393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.841350 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" podStartSLOduration=0.841334911 podStartE2EDuration="841.334911ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:28.838871794 +0000 UTC m=+1439.796488531" watchObservedRunningTime="2026-09-14 05:10:28.841334911 +0000 UTC m=+1439.798951647" Sep 14 05:10:28.853079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:28.853030 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" podStartSLOduration=0.853015698 podStartE2EDuration="853.015698ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:28.851724543 +0000 UTC m=+1439.809341281" watchObservedRunningTime="2026-09-14 05:10:28.853015698 +0000 UTC m=+1439.810632447" Sep 14 05:10:29.671660 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:29.671623 2581 scope.go:117] "RemoveContainer" containerID="a35acad051d5a868081dd71f490bd3583530d95ce77437768bc7ed591ca1e648" Sep 14 05:10:39.835060 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:39.835027 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:10:39.835530 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:39.835100 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:10:45.262573 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.262535 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:10:45.263004 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.262785 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" podUID="93bb0224-4827-4e75-95b2-90c358d40f72" containerName="payload-processing" containerID="cri-o://9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72" gracePeriod=30 Sep 14 05:10:45.537470 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.537363 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:10:45.686771 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.686733 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lz88\" (UniqueName: \"kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88\") pod \"93bb0224-4827-4e75-95b2-90c358d40f72\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " Sep 14 05:10:45.686961 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.686820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume\") pod \"93bb0224-4827-4e75-95b2-90c358d40f72\" (UID: \"93bb0224-4827-4e75-95b2-90c358d40f72\") " Sep 14 05:10:45.687237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.687211 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93bb0224-4827-4e75-95b2-90c358d40f72" (UID: "93bb0224-4827-4e75-95b2-90c358d40f72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:10:45.688823 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.688801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88" (OuterVolumeSpecName: "kube-api-access-5lz88") pod "93bb0224-4827-4e75-95b2-90c358d40f72" (UID: "93bb0224-4827-4e75-95b2-90c358d40f72"). InnerVolumeSpecName "kube-api-access-5lz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:10:45.788315 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.788237 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lz88\" (UniqueName: \"kubernetes.io/projected/93bb0224-4827-4e75-95b2-90c358d40f72-kube-api-access-5lz88\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:45.788315 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.788266 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bb0224-4827-4e75-95b2-90c358d40f72-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:10:45.917796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.917758 2581 generic.go:356] "Generic (PLEG): container finished" podID="93bb0224-4827-4e75-95b2-90c358d40f72" containerID="9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72" exitCode=0 Sep 14 05:10:45.917954 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.917821 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" Sep 14 05:10:45.917954 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.917842 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" event={"ID":"93bb0224-4827-4e75-95b2-90c358d40f72","Type":"ContainerDied","Data":"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72"} Sep 14 05:10:45.917954 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.917924 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5" event={"ID":"93bb0224-4827-4e75-95b2-90c358d40f72","Type":"ContainerDied","Data":"6bc8c9ac33af5bac117d728cbd36b9547a4c6121cba65997a4ce6ec2770d6fe2"} Sep 14 05:10:45.917954 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.917949 2581 scope.go:117] "RemoveContainer" containerID="9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72" Sep 14 05:10:45.927538 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.927518 2581 scope.go:117] "RemoveContainer" containerID="9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72" Sep 14 05:10:45.927817 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:10:45.927796 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72\": container with ID starting with 9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72 not found: ID does not exist" containerID="9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72" Sep 14 05:10:45.927883 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.927829 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72"} err="failed to get container status \"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72\": rpc error: code = NotFound desc = could not find container \"9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72\": container with ID starting with 9852821a2aaab6662a0dca82c97375810f01140f7e119ee37682124154e36e72 not found: ID does not exist" Sep 14 05:10:45.938000 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.937978 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:10:45.940032 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:45.940011 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c6a48c-787f9896f7-sbtt5"] Sep 14 05:10:47.553286 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:47.553246 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93bb0224-4827-4e75-95b2-90c358d40f72" path="/var/lib/kubelet/pods/93bb0224-4827-4e75-95b2-90c358d40f72/volumes" Sep 14 05:10:57.413177 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.413148 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:10:57.413716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.413586 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bb0224-4827-4e75-95b2-90c358d40f72" containerName="payload-processing" Sep 14 05:10:57.413716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.413599 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bb0224-4827-4e75-95b2-90c358d40f72" containerName="payload-processing" Sep 14 05:10:57.413716 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.413688 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bb0224-4827-4e75-95b2-90c358d40f72" containerName="payload-processing" Sep 14 05:10:57.418270 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.418250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.419980 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.419958 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-27be50\"" Sep 14 05:10:57.419980 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.419966 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-27be50-dockercfg-zdr7l\"" Sep 14 05:10:57.425151 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.425129 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:10:57.441424 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.441399 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:10:57.445736 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.445717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.454393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.454370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:10:57.500896 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.500868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.501005 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.500915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzt7p\" (UniqueName: \"kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.501005 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.500984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4r4\" (UniqueName: \"kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.501084 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.501009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.602182 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzt7p\" (UniqueName: \"kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.602345 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4r4\" (UniqueName: \"kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.602345 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.602345 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.602907 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602887 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.602977 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.602944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.610357 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.610330 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzt7p\" (UniqueName: \"kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p\") pod \"payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:57.610464 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.610402 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4r4\" (UniqueName: \"kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.730021 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.729966 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:57.757251 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:57.757220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:58.006767 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.006740 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:10:58.009127 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:10:58.009092 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb6493b_567a_4964_b3c5_ffb2bc92fd66.slice/crio-55926a55aa70aa1770c1c0a55762d1771e156fea0604c7c790738b9d3a8eee4e WatchSource:0}: Error finding container 55926a55aa70aa1770c1c0a55762d1771e156fea0604c7c790738b9d3a8eee4e: Status 404 returned error can't find the container with id 55926a55aa70aa1770c1c0a55762d1771e156fea0604c7c790738b9d3a8eee4e Sep 14 05:10:58.039081 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.039030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:10:58.050160 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:10:58.050131 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49527fdb_f6f0_4da6_b6c0_edb2f9d56099.slice/crio-3d3bab440e1666601ce910ec8788eafbdb1d3b7c4c66bab90920b18e139a37e2 WatchSource:0}: Error finding container 3d3bab440e1666601ce910ec8788eafbdb1d3b7c4c66bab90920b18e139a37e2: Status 404 returned error can't find the container with id 3d3bab440e1666601ce910ec8788eafbdb1d3b7c4c66bab90920b18e139a37e2 Sep 14 05:10:58.994920 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.994883 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" event={"ID":"49527fdb-f6f0-4da6-b6c0-edb2f9d56099","Type":"ContainerStarted","Data":"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c"} Sep 14 05:10:58.995374 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.994944 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" event={"ID":"49527fdb-f6f0-4da6-b6c0-edb2f9d56099","Type":"ContainerStarted","Data":"3d3bab440e1666601ce910ec8788eafbdb1d3b7c4c66bab90920b18e139a37e2"} Sep 14 05:10:58.995374 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.994973 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:10:58.996682 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.996650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" event={"ID":"8cb6493b-567a-4964-b3c5-ffb2bc92fd66","Type":"ContainerStarted","Data":"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab"} Sep 14 05:10:58.996793 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.996698 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" event={"ID":"8cb6493b-567a-4964-b3c5-ffb2bc92fd66","Type":"ContainerStarted","Data":"55926a55aa70aa1770c1c0a55762d1771e156fea0604c7c790738b9d3a8eee4e"} Sep 14 05:10:58.996793 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:58.996720 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:10:59.010974 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:59.010925 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" podStartSLOduration=2.010911539 podStartE2EDuration="2.010911539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:59.008856071 +0000 UTC m=+1469.966472808" watchObservedRunningTime="2026-09-14 05:10:59.010911539 +0000 UTC m=+1469.968528275" Sep 14 05:10:59.025156 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:10:59.025094 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" podStartSLOduration=2.02508276 podStartE2EDuration="2.02508276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:59.022954882 +0000 UTC m=+1469.980571620" watchObservedRunningTime="2026-09-14 05:10:59.02508276 +0000 UTC m=+1469.982699497" Sep 14 05:11:10.004910 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:11:10.004881 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:11:10.005358 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:11:10.004951 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:11:29.717728 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:11:29.717699 2581 scope.go:117] "RemoveContainer" containerID="a9bb33aa8e4e493930dc6d86a82982b86b180039c4e3d25e9a4da36888de8e95" Sep 14 05:11:29.726061 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:11:29.726039 2581 scope.go:117] "RemoveContainer" containerID="1ec314d064c803f3e1d98556fc894f7052f34459ce7ead560fc47d6a44956a9c" Sep 14 05:13:06.430852 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.430819 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp"] Sep 14 05:13:06.434337 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.434321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:06.436176 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.436153 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:13:06.436304 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.436181 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:13:06.440562 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.440539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp"] Sep 14 05:13:06.537634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.537593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp\" (UID: \"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:06.638998 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.638958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp\" (UID: \"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:06.639629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.639609 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp\" (UID: \"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:06.746677 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:06.746596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:07.017670 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:07.017646 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp"] Sep 14 05:13:07.019847 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:13:07.019816 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb9d5ad_ed60_49e6_996f_9b65871cf6bb.slice/crio-62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88 WatchSource:0}: Error finding container 62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88: Status 404 returned error can't find the container with id 62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88 Sep 14 05:13:07.699014 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:07.698976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" event={"ID":"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb","Type":"ContainerStarted","Data":"4786e976f0c92e40e2e4baa05bd1adfeb8e17a17c245cf1d5ceabf6e4a54a5bb"} Sep 14 05:13:07.699481 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:07.699034 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" event={"ID":"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb","Type":"ContainerStarted","Data":"62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88"} Sep 14 05:13:07.738987 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:07.738932 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" podStartSLOduration=1.738910888 podStartE2EDuration="1.738910888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:07.737699772 +0000 UTC m=+1598.695316509" watchObservedRunningTime="2026-09-14 05:13:07.738910888 +0000 UTC m=+1598.696527626" Sep 14 05:13:08.706131 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:08.706089 2581 generic.go:356] "Generic (PLEG): container finished" podID="9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" containerID="4786e976f0c92e40e2e4baa05bd1adfeb8e17a17c245cf1d5ceabf6e4a54a5bb" exitCode=0 Sep 14 05:13:08.706527 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:08.706145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" event={"ID":"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb","Type":"ContainerDied","Data":"4786e976f0c92e40e2e4baa05bd1adfeb8e17a17c245cf1d5ceabf6e4a54a5bb"} Sep 14 05:13:09.856656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:09.856632 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:09.974841 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:09.974755 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca\") pod \"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb\" (UID: \"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb\") " Sep 14 05:13:09.975188 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:09.975166 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" (UID: "9bb9d5ad-ed60-49e6-996f-9b65871cf6bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:13:10.076198 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:10.076163 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:10.720449 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:10.720417 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" Sep 14 05:13:10.720628 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:10.720420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp" event={"ID":"9bb9d5ad-ed60-49e6-996f-9b65871cf6bb","Type":"ContainerDied","Data":"62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88"} Sep 14 05:13:10.720628 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:10.720536 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62789ad2b20fe12f4e4f5ee2145038ad1999800fcdeedc2cee710432a0b4fe88" Sep 14 05:13:12.386611 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.386578 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:13:12.387142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.387107 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" containerName="revoke-keys" Sep 14 05:13:12.387142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.387136 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" containerName="revoke-keys" Sep 14 05:13:12.387237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.387216 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" containerName="revoke-keys" Sep 14 05:13:12.390453 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.390438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.392093 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.392070 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6cb472\"" Sep 14 05:13:12.399404 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.399382 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:13:12.499082 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.499051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwfs\" (UniqueName: \"kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.499262 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.499162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.600565 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.600524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwfs\" (UniqueName: \"kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.600755 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.600633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.601400 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.601379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.607662 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.607633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwfs\" (UniqueName: \"kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs\") pod \"payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.702155 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.702046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:12.976311 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:12.976282 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:13:12.978595 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:13:12.978567 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df43b1b_9f63_43c9_9ae1_3ec2fd211b4b.slice/crio-ab89336c8aee646e20d0ef26f8607397e8f6e50e6b9e64535ca274b803dbb7bd WatchSource:0}: Error finding container ab89336c8aee646e20d0ef26f8607397e8f6e50e6b9e64535ca274b803dbb7bd: Status 404 returned error can't find the container with id ab89336c8aee646e20d0ef26f8607397e8f6e50e6b9e64535ca274b803dbb7bd Sep 14 05:13:13.744828 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:13.744785 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" event={"ID":"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b","Type":"ContainerStarted","Data":"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a"} Sep 14 05:13:13.745264 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:13.744860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" event={"ID":"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b","Type":"ContainerStarted","Data":"ab89336c8aee646e20d0ef26f8607397e8f6e50e6b9e64535ca274b803dbb7bd"} Sep 14 05:13:13.745264 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:13.744935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:13.760633 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:13.760578 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" podStartSLOduration=1.760559254 podStartE2EDuration="1.760559254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:13.759485186 +0000 UTC m=+1604.717101950" watchObservedRunningTime="2026-09-14 05:13:13.760559254 +0000 UTC m=+1604.718175997" Sep 14 05:13:22.272970 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.272880 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:13:22.276753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.276714 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" podUID="8bd771d8-d761-471a-aa98-2bb0110c535a" containerName="payload-processing" containerID="cri-o://9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537" gracePeriod=30 Sep 14 05:13:22.278166 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.278138 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:13:22.278381 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.278358 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" podUID="a820576f-886e-4d86-ac7d-88ae43ca9387" containerName="payload-pre-processing" containerID="cri-o://391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2" gracePeriod=30 Sep 14 05:13:22.446094 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.446038 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp"] Sep 14 05:13:22.449700 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.449673 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e08402-gsjdp"] Sep 14 05:13:22.559148 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.558852 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:13:22.600689 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.600664 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:13:22.603362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.603335 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbww\" (UniqueName: \"kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww\") pod \"8bd771d8-d761-471a-aa98-2bb0110c535a\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " Sep 14 05:13:22.603513 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.603380 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume\") pod \"8bd771d8-d761-471a-aa98-2bb0110c535a\" (UID: \"8bd771d8-d761-471a-aa98-2bb0110c535a\") " Sep 14 05:13:22.603837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.603813 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bd771d8-d761-471a-aa98-2bb0110c535a" (UID: "8bd771d8-d761-471a-aa98-2bb0110c535a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:13:22.604236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.604213 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd771d8-d761-471a-aa98-2bb0110c535a-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:22.605634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.605610 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww" (OuterVolumeSpecName: "kube-api-access-6cbww") pod "8bd771d8-d761-471a-aa98-2bb0110c535a" (UID: "8bd771d8-d761-471a-aa98-2bb0110c535a"). InnerVolumeSpecName "kube-api-access-6cbww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:13:22.705442 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.705410 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume\") pod \"a820576f-886e-4d86-ac7d-88ae43ca9387\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " Sep 14 05:13:22.705619 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.705478 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz4mz\" (UniqueName: \"kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz\") pod \"a820576f-886e-4d86-ac7d-88ae43ca9387\" (UID: \"a820576f-886e-4d86-ac7d-88ae43ca9387\") " Sep 14 05:13:22.705725 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.705698 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbww\" (UniqueName: \"kubernetes.io/projected/8bd771d8-d761-471a-aa98-2bb0110c535a-kube-api-access-6cbww\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:22.705793 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.705745 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a820576f-886e-4d86-ac7d-88ae43ca9387" (UID: "a820576f-886e-4d86-ac7d-88ae43ca9387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:13:22.707530 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.707508 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz" (OuterVolumeSpecName: "kube-api-access-zz4mz") pod "a820576f-886e-4d86-ac7d-88ae43ca9387" (UID: "a820576f-886e-4d86-ac7d-88ae43ca9387"). InnerVolumeSpecName "kube-api-access-zz4mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:13:22.795680 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.795606 2581 generic.go:356] "Generic (PLEG): container finished" podID="8bd771d8-d761-471a-aa98-2bb0110c535a" containerID="9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537" exitCode=0 Sep 14 05:13:22.795680 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.795658 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" Sep 14 05:13:22.795853 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.795692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" event={"ID":"8bd771d8-d761-471a-aa98-2bb0110c535a","Type":"ContainerDied","Data":"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537"} Sep 14 05:13:22.795853 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.795749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq" event={"ID":"8bd771d8-d761-471a-aa98-2bb0110c535a","Type":"ContainerDied","Data":"5ee15603f1606a45950c5315f0f8516a9dbb989cf1039be15f6ad52a96b11c83"} Sep 14 05:13:22.795853 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.795780 2581 scope.go:117] "RemoveContainer" containerID="9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537" Sep 14 05:13:22.797083 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.797057 2581 generic.go:356] "Generic (PLEG): container finished" podID="a820576f-886e-4d86-ac7d-88ae43ca9387" containerID="391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2" exitCode=0 Sep 14 05:13:22.797225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.797091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" event={"ID":"a820576f-886e-4d86-ac7d-88ae43ca9387","Type":"ContainerDied","Data":"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2"} Sep 14 05:13:22.797225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.797151 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" event={"ID":"a820576f-886e-4d86-ac7d-88ae43ca9387","Type":"ContainerDied","Data":"7e77f0a4823ec2dd2c390efc591f051479fb07273d81f3005e12b7f18f3cb033"} Sep 14 05:13:22.797225 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.797095 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh" Sep 14 05:13:22.807669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.807170 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a820576f-886e-4d86-ac7d-88ae43ca9387-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:22.807669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.807192 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4mz\" (UniqueName: \"kubernetes.io/projected/a820576f-886e-4d86-ac7d-88ae43ca9387-kube-api-access-zz4mz\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:22.807669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.807656 2581 scope.go:117] "RemoveContainer" containerID="9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537" Sep 14 05:13:22.808317 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:13:22.808297 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537\": container with ID starting with 9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537 not found: ID does not exist" containerID="9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537" Sep 14 05:13:22.808418 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.808325 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537"} err="failed to get container status \"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537\": rpc error: code = NotFound desc = could not find container \"9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537\": container with ID starting with 9b019414f3f4cf05b5ece0c9bc5fd9cff930c9a4e8bac621b7114c6a79915537 not found: ID does not exist" Sep 14 05:13:22.808418 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.808341 2581 scope.go:117] "RemoveContainer" containerID="391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2" Sep 14 05:13:22.816682 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.816663 2581 scope.go:117] "RemoveContainer" containerID="391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2" Sep 14 05:13:22.816916 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:13:22.816901 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2\": container with ID starting with 391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2 not found: ID does not exist" containerID="391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2" Sep 14 05:13:22.816963 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.816921 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2"} err="failed to get container status \"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2\": rpc error: code = NotFound desc = could not find container \"391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2\": container with ID starting with 391de3191de25ad297ec93f0d5cb28c12d66b72be5c4615ee12a297b8cb87ed2 not found: ID does not exist" Sep 14 05:13:22.821625 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.821605 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:13:22.825394 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.825374 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e08402-67784fb846-dhznh"] Sep 14 05:13:22.833428 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.833408 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:13:22.836630 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:22.836603 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e08402-7964d8d469-6tppq"] Sep 14 05:13:23.554199 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:23.554097 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd771d8-d761-471a-aa98-2bb0110c535a" path="/var/lib/kubelet/pods/8bd771d8-d761-471a-aa98-2bb0110c535a/volumes" Sep 14 05:13:23.555303 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:23.555281 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb9d5ad-ed60-49e6-996f-9b65871cf6bb" path="/var/lib/kubelet/pods/9bb9d5ad-ed60-49e6-996f-9b65871cf6bb/volumes" Sep 14 05:13:23.555969 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:23.555950 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a820576f-886e-4d86-ac7d-88ae43ca9387" path="/var/lib/kubelet/pods/a820576f-886e-4d86-ac7d-88ae43ca9387/volumes" Sep 14 05:13:24.753276 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:24.753241 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:13:29.763448 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:29.763412 2581 scope.go:117] "RemoveContainer" containerID="cc4490bacf807d99334f592d41250ac2b24778555e937d5979f64e56d2bf0ec3" Sep 14 05:13:30.107504 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:30.107465 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:13:30.107773 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:30.107723 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" containerID="cri-o://921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5" gracePeriod=30 Sep 14 05:13:35.393096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.393068 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:13:35.449018 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.448986 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449074 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449099 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449155 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g68lr\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449183 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449427 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449242 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449427 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449295 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449427 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449341 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449427 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449389 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket\") pod \"096381bd-4027-4dbd-93c8-1747155bf0cf\" (UID: \"096381bd-4027-4dbd-93c8-1747155bf0cf\") " Sep 14 05:13:35.449427 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449397 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:13:35.449619 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449505 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:13:35.449724 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449701 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:13:35.449724 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449715 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:13:35.449873 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449856 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.449944 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449878 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.449944 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449891 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.449944 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.449905 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.450425 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.450394 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:13:35.451574 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.451544 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr" (OuterVolumeSpecName: "kube-api-access-g68lr") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "kube-api-access-g68lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:13:35.451725 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.451707 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:13:35.451788 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.451726 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:13:35.451788 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.451739 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "096381bd-4027-4dbd-93c8-1747155bf0cf" (UID: "096381bd-4027-4dbd-93c8-1747155bf0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:13:35.550463 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.550433 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.550463 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.550455 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g68lr\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-kube-api-access-g68lr\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.550463 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.550466 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.550694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.550474 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096381bd-4027-4dbd-93c8-1747155bf0cf-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.550694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.550482 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096381bd-4027-4dbd-93c8-1747155bf0cf-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:13:35.882905 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.882878 2581 generic.go:356] "Generic (PLEG): container finished" podID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerID="921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5" exitCode=0 Sep 14 05:13:35.883058 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.882927 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" Sep 14 05:13:35.883058 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.882959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerDied","Data":"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5"} Sep 14 05:13:35.883058 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.883026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r" event={"ID":"096381bd-4027-4dbd-93c8-1747155bf0cf","Type":"ContainerDied","Data":"d056e0bdccf27b98d46472ecac8f54601c392fdfa024a33affc94199f5027e7c"} Sep 14 05:13:35.883058 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.883054 2581 scope.go:117] "RemoveContainer" containerID="921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5" Sep 14 05:13:35.892079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.892062 2581 scope.go:117] "RemoveContainer" containerID="5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3" Sep 14 05:13:35.898753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.898729 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:13:35.902059 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.902041 2581 scope.go:117] "RemoveContainer" containerID="921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5" Sep 14 05:13:35.902435 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:13:35.902415 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5\": container with ID starting with 921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5 not found: ID does not exist" containerID="921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5" Sep 14 05:13:35.902562 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.902446 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5"} err="failed to get container status \"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5\": rpc error: code = NotFound desc = could not find container \"921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5\": container with ID starting with 921646c1104219e22b9e4a52e6a4b749ed36830d2b9324ed91d98ded6a6e7aa5 not found: ID does not exist" Sep 14 05:13:35.902562 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.902465 2581 scope.go:117] "RemoveContainer" containerID="5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3" Sep 14 05:13:35.902766 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:13:35.902746 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3\": container with ID starting with 5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3 not found: ID does not exist" containerID="5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3" Sep 14 05:13:35.902826 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.902777 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3"} err="failed to get container status \"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3\": rpc error: code = NotFound desc = could not find container \"5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3\": container with ID starting with 5efccd0cb9f1ec0077989d10c2884f5832ff339c4e84de6cd3523bedbfd6c5e3 not found: ID does not exist" Sep 14 05:13:35.904246 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:35.904225 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e08402-openshift-default-f6f5dc8f4-czx5r"] Sep 14 05:13:37.293683 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.293650 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294129 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd771d8-d761-471a-aa98-2bb0110c535a" containerName="payload-processing" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294142 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd771d8-d761-471a-aa98-2bb0110c535a" containerName="payload-processing" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294158 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294167 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294181 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a820576f-886e-4d86-ac7d-88ae43ca9387" containerName="payload-pre-processing" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294187 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a820576f-886e-4d86-ac7d-88ae43ca9387" containerName="payload-pre-processing" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294196 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.294218 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294201 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.294516 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294280 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd771d8-d761-471a-aa98-2bb0110c535a" containerName="payload-processing" Sep 14 05:13:37.294516 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294290 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.294516 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.294300 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a820576f-886e-4d86-ac7d-88ae43ca9387" containerName="payload-pre-processing" Sep 14 05:13:37.298580 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.298557 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.300470 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.300443 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-09203c-dockercfg-29wnc\"" Sep 14 05:13:37.300590 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.300488 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-09203c\"" Sep 14 05:13:37.306847 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.306733 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:13:37.370203 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.370174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfh89\" (UniqueName: \"kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.370330 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.370227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.431084 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.431048 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:13:37.431595 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.431580 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" containerName="istio-proxy" Sep 14 05:13:37.434781 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.434765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.444036 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.444016 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:13:37.471629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.471601 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfh89\" (UniqueName: \"kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.471776 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.471650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.472301 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.472282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.481413 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.481392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfh89\" (UniqueName: \"kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89\") pod \"payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.552924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.552856 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096381bd-4027-4dbd-93c8-1747155bf0cf" path="/var/lib/kubelet/pods/096381bd-4027-4dbd-93c8-1747155bf0cf/volumes" Sep 14 05:13:37.572491 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.572467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9xd\" (UniqueName: \"kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.572629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.572550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.610296 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.610266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:37.674153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.673849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9xd\" (UniqueName: \"kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.674153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.673934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.674658 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.674634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.681691 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.681659 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9xd\" (UniqueName: \"kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd\") pod \"payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.746204 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.746167 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:37.881956 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.881930 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:13:37.884112 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:13:37.884085 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af3bafe_98b6_4901_a698_752eabb7c2d5.slice/crio-647ff2c4678b0d4e7d6026226cbbb2f10c973e42102707618910279d9cb357e0 WatchSource:0}: Error finding container 647ff2c4678b0d4e7d6026226cbbb2f10c973e42102707618910279d9cb357e0: Status 404 returned error can't find the container with id 647ff2c4678b0d4e7d6026226cbbb2f10c973e42102707618910279d9cb357e0 Sep 14 05:13:37.897983 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:37.897891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" event={"ID":"5af3bafe-98b6-4901-a698-752eabb7c2d5","Type":"ContainerStarted","Data":"647ff2c4678b0d4e7d6026226cbbb2f10c973e42102707618910279d9cb357e0"} Sep 14 05:13:38.019864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.019836 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:13:38.021927 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:13:38.021896 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb09f956_ad4e_4a6b_b345_69de94266b36.slice/crio-1c11ca9fab713f6599540c4bd39de041aae0a7edde453e77e3ddf8b2b0bcdea4 WatchSource:0}: Error finding container 1c11ca9fab713f6599540c4bd39de041aae0a7edde453e77e3ddf8b2b0bcdea4: Status 404 returned error can't find the container with id 1c11ca9fab713f6599540c4bd39de041aae0a7edde453e77e3ddf8b2b0bcdea4 Sep 14 05:13:38.904900 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.904856 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" event={"ID":"cb09f956-ad4e-4a6b-b345-69de94266b36","Type":"ContainerStarted","Data":"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781"} Sep 14 05:13:38.905380 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.905043 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:38.905380 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.905093 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" event={"ID":"cb09f956-ad4e-4a6b-b345-69de94266b36","Type":"ContainerStarted","Data":"1c11ca9fab713f6599540c4bd39de041aae0a7edde453e77e3ddf8b2b0bcdea4"} Sep 14 05:13:38.906675 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.906648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" event={"ID":"5af3bafe-98b6-4901-a698-752eabb7c2d5","Type":"ContainerStarted","Data":"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966"} Sep 14 05:13:38.906802 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.906699 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:13:38.921381 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.921337 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" podStartSLOduration=1.921324153 podStartE2EDuration="1.921324153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:38.918636877 +0000 UTC m=+1629.876253614" watchObservedRunningTime="2026-09-14 05:13:38.921324153 +0000 UTC m=+1629.878940932" Sep 14 05:13:38.932551 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:38.932506 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" podStartSLOduration=1.932494276 podStartE2EDuration="1.932494276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:38.931036405 +0000 UTC m=+1629.888653141" watchObservedRunningTime="2026-09-14 05:13:38.932494276 +0000 UTC m=+1629.890111013" Sep 14 05:13:49.913770 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:49.913737 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:13:49.914164 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:13:49.913802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:14:21.559920 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.559884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:14:21.560485 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.560199 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" podUID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" containerName="payload-pre-processing" containerID="cri-o://243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a" gracePeriod=30 Sep 14 05:14:21.813975 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.813912 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:14:21.904242 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.904204 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rwfs\" (UniqueName: \"kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs\") pod \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " Sep 14 05:14:21.904454 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.904252 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume\") pod \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\" (UID: \"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b\") " Sep 14 05:14:21.904978 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.904942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" (UID: "7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:14:21.907433 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:21.907391 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs" (OuterVolumeSpecName: "kube-api-access-9rwfs") pod "7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" (UID: "7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b"). InnerVolumeSpecName "kube-api-access-9rwfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:14:22.005729 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.005698 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwfs\" (UniqueName: \"kubernetes.io/projected/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-kube-api-access-9rwfs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:22.005729 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.005726 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:22.154236 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.154202 2581 generic.go:356] "Generic (PLEG): container finished" podID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" containerID="243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a" exitCode=0 Sep 14 05:14:22.154391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.154252 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" Sep 14 05:14:22.154391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.154285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" event={"ID":"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b","Type":"ContainerDied","Data":"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a"} Sep 14 05:14:22.154391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.154339 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s" event={"ID":"7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b","Type":"ContainerDied","Data":"ab89336c8aee646e20d0ef26f8607397e8f6e50e6b9e64535ca274b803dbb7bd"} Sep 14 05:14:22.154391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.154364 2581 scope.go:117] "RemoveContainer" containerID="243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a" Sep 14 05:14:22.163700 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.163682 2581 scope.go:117] "RemoveContainer" containerID="243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a" Sep 14 05:14:22.163967 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:14:22.163949 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a\": container with ID starting with 243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a not found: ID does not exist" containerID="243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a" Sep 14 05:14:22.164028 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.163974 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a"} err="failed to get container status \"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a\": rpc error: code = NotFound desc = could not find container \"243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a\": container with ID starting with 243b74bb64285f0570fb0bb96ced0e0fa5177f2c57795937ba509bb71eee811a not found: ID does not exist" Sep 14 05:14:22.173728 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.173704 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:14:22.177264 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:22.177242 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cb472-7b66768854-pk99s"] Sep 14 05:14:23.552943 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:23.552909 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" path="/var/lib/kubelet/pods/7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b/volumes" Sep 14 05:14:29.792321 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:29.792290 2581 scope.go:117] "RemoveContainer" containerID="421fc825a39dcac79a1c2f4b7d663a68f769f11298d5463bcdcf2b611e2c1c4f" Sep 14 05:14:44.338386 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.338342 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:14:44.338918 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.338643 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" podUID="cb09f956-ad4e-4a6b-b345-69de94266b36" containerName="payload-processing" containerID="cri-o://fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781" gracePeriod=30 Sep 14 05:14:44.341554 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.341525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:14:44.341800 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.341762 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" podUID="5af3bafe-98b6-4901-a698-752eabb7c2d5" containerName="payload-pre-processing" containerID="cri-o://a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966" gracePeriod=30 Sep 14 05:14:44.639852 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.639825 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:14:44.670675 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.670643 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:14:44.739530 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.739494 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume\") pod \"5af3bafe-98b6-4901-a698-752eabb7c2d5\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " Sep 14 05:14:44.739720 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.739650 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq9xd\" (UniqueName: \"kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd\") pod \"cb09f956-ad4e-4a6b-b345-69de94266b36\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " Sep 14 05:14:44.739720 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.739702 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume\") pod \"cb09f956-ad4e-4a6b-b345-69de94266b36\" (UID: \"cb09f956-ad4e-4a6b-b345-69de94266b36\") " Sep 14 05:14:44.739837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.739742 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfh89\" (UniqueName: \"kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89\") pod \"5af3bafe-98b6-4901-a698-752eabb7c2d5\" (UID: \"5af3bafe-98b6-4901-a698-752eabb7c2d5\") " Sep 14 05:14:44.740083 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.740053 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5af3bafe-98b6-4901-a698-752eabb7c2d5" (UID: "5af3bafe-98b6-4901-a698-752eabb7c2d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:14:44.740179 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.740077 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb09f956-ad4e-4a6b-b345-69de94266b36" (UID: "cb09f956-ad4e-4a6b-b345-69de94266b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:14:44.741829 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.741804 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd" (OuterVolumeSpecName: "kube-api-access-xq9xd") pod "cb09f956-ad4e-4a6b-b345-69de94266b36" (UID: "cb09f956-ad4e-4a6b-b345-69de94266b36"). InnerVolumeSpecName "kube-api-access-xq9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:14:44.741897 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.741865 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89" (OuterVolumeSpecName: "kube-api-access-nfh89") pod "5af3bafe-98b6-4901-a698-752eabb7c2d5" (UID: "5af3bafe-98b6-4901-a698-752eabb7c2d5"). InnerVolumeSpecName "kube-api-access-nfh89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:14:44.841140 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.841083 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9xd\" (UniqueName: \"kubernetes.io/projected/cb09f956-ad4e-4a6b-b345-69de94266b36-kube-api-access-xq9xd\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:44.841140 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.841145 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb09f956-ad4e-4a6b-b345-69de94266b36-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:44.841342 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.841156 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfh89\" (UniqueName: \"kubernetes.io/projected/5af3bafe-98b6-4901-a698-752eabb7c2d5-kube-api-access-nfh89\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:44.841342 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:44.841166 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af3bafe-98b6-4901-a698-752eabb7c2d5-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:45.285461 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.285425 2581 generic.go:356] "Generic (PLEG): container finished" podID="5af3bafe-98b6-4901-a698-752eabb7c2d5" containerID="a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966" exitCode=0 Sep 14 05:14:45.285651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.285485 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" Sep 14 05:14:45.285651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.285515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" event={"ID":"5af3bafe-98b6-4901-a698-752eabb7c2d5","Type":"ContainerDied","Data":"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966"} Sep 14 05:14:45.285651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.285601 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7" event={"ID":"5af3bafe-98b6-4901-a698-752eabb7c2d5","Type":"ContainerDied","Data":"647ff2c4678b0d4e7d6026226cbbb2f10c973e42102707618910279d9cb357e0"} Sep 14 05:14:45.285651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.285635 2581 scope.go:117] "RemoveContainer" containerID="a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966" Sep 14 05:14:45.287015 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.286990 2581 generic.go:356] "Generic (PLEG): container finished" podID="cb09f956-ad4e-4a6b-b345-69de94266b36" containerID="fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781" exitCode=0 Sep 14 05:14:45.287153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.287017 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" event={"ID":"cb09f956-ad4e-4a6b-b345-69de94266b36","Type":"ContainerDied","Data":"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781"} Sep 14 05:14:45.287153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.287053 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" Sep 14 05:14:45.287153 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.287058 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w" event={"ID":"cb09f956-ad4e-4a6b-b345-69de94266b36","Type":"ContainerDied","Data":"1c11ca9fab713f6599540c4bd39de041aae0a7edde453e77e3ddf8b2b0bcdea4"} Sep 14 05:14:45.295142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.295100 2581 scope.go:117] "RemoveContainer" containerID="a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966" Sep 14 05:14:45.295567 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:14:45.295538 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966\": container with ID starting with a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966 not found: ID does not exist" containerID="a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966" Sep 14 05:14:45.295641 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.295578 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966"} err="failed to get container status \"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966\": rpc error: code = NotFound desc = could not find container \"a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966\": container with ID starting with a6dfe64ed411bc4ed5392e04009e9e52bacba3ed6e5717d8d4c35486a79d2966 not found: ID does not exist" Sep 14 05:14:45.295641 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.295601 2581 scope.go:117] "RemoveContainer" containerID="fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781" Sep 14 05:14:45.304275 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.304257 2581 scope.go:117] "RemoveContainer" containerID="fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781" Sep 14 05:14:45.304525 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:14:45.304506 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781\": container with ID starting with fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781 not found: ID does not exist" containerID="fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781" Sep 14 05:14:45.304573 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.304530 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781"} err="failed to get container status \"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781\": rpc error: code = NotFound desc = could not find container \"fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781\": container with ID starting with fa3dc803ddb4523346a09137405e1161449aa8fcd566bfe61b0783021a567781 not found: ID does not exist" Sep 14 05:14:45.309538 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.309518 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:14:45.312904 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.312880 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-09203c-54db9f8bc9-gww9w"] Sep 14 05:14:45.320745 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.320723 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:14:45.326265 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.326242 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-09203c-c76dd9494-b95q7"] Sep 14 05:14:45.553391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.553309 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af3bafe-98b6-4901-a698-752eabb7c2d5" path="/var/lib/kubelet/pods/5af3bafe-98b6-4901-a698-752eabb7c2d5/volumes" Sep 14 05:14:45.553732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:45.553673 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb09f956-ad4e-4a6b-b345-69de94266b36" path="/var/lib/kubelet/pods/cb09f956-ad4e-4a6b-b345-69de94266b36/volumes" Sep 14 05:14:51.409439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409395 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6"] Sep 14 05:14:51.409914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409879 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" containerName="payload-pre-processing" Sep 14 05:14:51.409914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409892 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" containerName="payload-pre-processing" Sep 14 05:14:51.409914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409900 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af3bafe-98b6-4901-a698-752eabb7c2d5" containerName="payload-pre-processing" Sep 14 05:14:51.409914 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409906 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af3bafe-98b6-4901-a698-752eabb7c2d5" containerName="payload-pre-processing" Sep 14 05:14:51.410110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409926 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb09f956-ad4e-4a6b-b345-69de94266b36" containerName="payload-processing" Sep 14 05:14:51.410110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.409932 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb09f956-ad4e-4a6b-b345-69de94266b36" containerName="payload-processing" Sep 14 05:14:51.410110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.410002 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb09f956-ad4e-4a6b-b345-69de94266b36" containerName="payload-processing" Sep 14 05:14:51.410110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.410013 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df43b1b-9f63-43c9-9ae1-3ec2fd211b4b" containerName="payload-pre-processing" Sep 14 05:14:51.410110 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.410019 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af3bafe-98b6-4901-a698-752eabb7c2d5" containerName="payload-pre-processing" Sep 14 05:14:51.414742 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.414724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:51.416549 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.416525 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:14:51.416661 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.416548 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:14:51.420176 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.420151 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6"] Sep 14 05:14:51.511179 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.511104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6\" (UID: \"395552f4-00ef-40d6-be81-e1ece10d310c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:51.612598 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.612559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6\" (UID: \"395552f4-00ef-40d6-be81-e1ece10d310c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:51.613307 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.613284 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6\" (UID: \"395552f4-00ef-40d6-be81-e1ece10d310c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:51.728037 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:51.727939 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:52.004505 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:52.004375 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6"] Sep 14 05:14:52.013696 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:14:52.013616 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395552f4_00ef_40d6_be81_e1ece10d310c.slice/crio-d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f WatchSource:0}: Error finding container d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f: Status 404 returned error can't find the container with id d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f Sep 14 05:14:52.329890 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:52.329802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" event={"ID":"395552f4-00ef-40d6-be81-e1ece10d310c","Type":"ContainerStarted","Data":"d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f"} Sep 14 05:14:53.336758 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:53.336728 2581 generic.go:356] "Generic (PLEG): container finished" podID="395552f4-00ef-40d6-be81-e1ece10d310c" containerID="8a55a8b177e87428afa22785698f4c6184a153f08ef5a873a1b7213872f6a35f" exitCode=0 Sep 14 05:14:53.337237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:53.336819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" event={"ID":"395552f4-00ef-40d6-be81-e1ece10d310c","Type":"ContainerDied","Data":"8a55a8b177e87428afa22785698f4c6184a153f08ef5a873a1b7213872f6a35f"} Sep 14 05:14:54.482350 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:54.482325 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:54.644201 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:54.644167 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca\") pod \"395552f4-00ef-40d6-be81-e1ece10d310c\" (UID: \"395552f4-00ef-40d6-be81-e1ece10d310c\") " Sep 14 05:14:54.644629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:54.644605 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "395552f4-00ef-40d6-be81-e1ece10d310c" (UID: "395552f4-00ef-40d6-be81-e1ece10d310c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:14:54.745709 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:54.745673 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/395552f4-00ef-40d6-be81-e1ece10d310c-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:14:55.349874 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:55.349837 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" Sep 14 05:14:55.349874 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:55.349861 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6" event={"ID":"395552f4-00ef-40d6-be81-e1ece10d310c","Type":"ContainerDied","Data":"d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f"} Sep 14 05:14:55.350173 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:14:55.349916 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d401c3fbd54fc3380ba2dce3be576bc8bcd878b939b6435e48682e8aad28010f" Sep 14 05:15:00.137692 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.137652 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:00.138112 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.138099 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395552f4-00ef-40d6-be81-e1ece10d310c" containerName="revoke-keys" Sep 14 05:15:00.138191 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.138140 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="395552f4-00ef-40d6-be81-e1ece10d310c" containerName="revoke-keys" Sep 14 05:15:00.138243 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.138233 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="395552f4-00ef-40d6-be81-e1ece10d310c" containerName="revoke-keys" Sep 14 05:15:00.141686 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.141668 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck"] Sep 14 05:15:00.141797 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.141776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" Sep 14 05:15:00.143524 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.143499 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:15:00.146231 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.146210 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" Sep 14 05:15:00.146957 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.146899 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:00.149835 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.149806 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck"] Sep 14 05:15:00.156210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.156186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" Sep 14 05:15:00.164663 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.164635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" Sep 14 05:15:00.439191 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.439161 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:00.440743 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:15:00.440722 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb3e55f_0c5a_4119_8c4a_74c052efb949.slice/crio-d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205 WatchSource:0}: Error finding container d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205: Status 404 returned error can't find the container with id d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205 Sep 14 05:15:00.444802 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:00.444774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck"] Sep 14 05:15:00.445804 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:15:00.445786 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31e50f0_7922_481a_b8e6_324411da33a2.slice/crio-24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb WatchSource:0}: Error finding container 24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb: Status 404 returned error can't find the container with id 24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb Sep 14 05:15:01.384680 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.384644 2581 generic.go:356] "Generic (PLEG): container finished" podID="b31e50f0-7922-481a-b8e6-324411da33a2" containerID="1f2220db74d631fabb04f874127c4c440912c2336fd2293f8ac6ae99f223c9fc" exitCode=0 Sep 14 05:15:01.385098 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.384735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" event={"ID":"b31e50f0-7922-481a-b8e6-324411da33a2","Type":"ContainerDied","Data":"1f2220db74d631fabb04f874127c4c440912c2336fd2293f8ac6ae99f223c9fc"} Sep 14 05:15:01.385098 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.384844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" event={"ID":"b31e50f0-7922-481a-b8e6-324411da33a2","Type":"ContainerStarted","Data":"24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb"} Sep 14 05:15:01.386372 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.386345 2581 generic.go:356] "Generic (PLEG): container finished" podID="adb3e55f-0c5a-4119-8c4a-74c052efb949" containerID="6043c9d2bd0549633e63d6887af7fbc2ba381781eae541a70837f7c438656242" exitCode=0 Sep 14 05:15:01.386499 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.386424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" event={"ID":"adb3e55f-0c5a-4119-8c4a-74c052efb949","Type":"ContainerDied","Data":"6043c9d2bd0549633e63d6887af7fbc2ba381781eae541a70837f7c438656242"} Sep 14 05:15:01.386499 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:01.386465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" event={"ID":"adb3e55f-0c5a-4119-8c4a-74c052efb949","Type":"ContainerStarted","Data":"d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205"} Sep 14 05:15:02.079724 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.079689 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:02.107554 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.107522 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:15:02.107755 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.107728 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" podUID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" containerName="payload-processing" containerID="cri-o://3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663" gracePeriod=30 Sep 14 05:15:02.275071 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.275037 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6"] Sep 14 05:15:02.277623 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.277594 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2eeb50-hdfm6"] Sep 14 05:15:02.377053 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.377028 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:15:02.394439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.394108 2581 generic.go:356] "Generic (PLEG): container finished" podID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" containerID="3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663" exitCode=0 Sep 14 05:15:02.394439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.394170 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" Sep 14 05:15:02.394439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.394181 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" event={"ID":"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36","Type":"ContainerDied","Data":"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663"} Sep 14 05:15:02.394439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.394309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq" event={"ID":"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36","Type":"ContainerDied","Data":"9edf2041a4bfaabfae1266bdfa41681ef40134dccd1108f9dd3644c5183e5787"} Sep 14 05:15:02.395043 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.394503 2581 scope.go:117] "RemoveContainer" containerID="3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663" Sep 14 05:15:02.409392 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.409281 2581 scope.go:117] "RemoveContainer" containerID="3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663" Sep 14 05:15:02.409729 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:15:02.409693 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663\": container with ID starting with 3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663 not found: ID does not exist" containerID="3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663" Sep 14 05:15:02.409729 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.409721 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663"} err="failed to get container status \"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663\": rpc error: code = NotFound desc = could not find container \"3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663\": container with ID starting with 3d14f5e15891846048322ddd00467388664f89fa1b4b8ea5519c706cd2d3b663 not found: ID does not exist" Sep 14 05:15:02.524277 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.524249 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume\") pod \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " Sep 14 05:15:02.524452 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.524302 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8grj\" (UniqueName: \"kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj\") pod \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\" (UID: \"3f0cc9c6-00d4-4b17-b212-5e8aac5eda36\") " Sep 14 05:15:02.524884 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.524842 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" (UID: "3f0cc9c6-00d4-4b17-b212-5e8aac5eda36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:15:02.527993 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.527932 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj" (OuterVolumeSpecName: "kube-api-access-c8grj") pod "3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" (UID: "3f0cc9c6-00d4-4b17-b212-5e8aac5eda36"). InnerVolumeSpecName "kube-api-access-c8grj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:15:02.555592 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.555571 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" Sep 14 05:15:02.567633 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.567616 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" Sep 14 05:15:02.625655 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.625631 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8grj\" (UniqueName: \"kubernetes.io/projected/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-kube-api-access-c8grj\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:02.625655 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.625655 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:02.716229 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.716202 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:15:02.723578 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:02.723555 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2eeb50-64f96b44bc-8wgzq"] Sep 14 05:15:03.402708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.402677 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" Sep 14 05:15:03.403174 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.402680 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck" event={"ID":"b31e50f0-7922-481a-b8e6-324411da33a2","Type":"ContainerDied","Data":"24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb"} Sep 14 05:15:03.403174 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.402768 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a1148dcd4fedca696980ba1fa559553869e8287f53bf75666265a0bd2de9fb" Sep 14 05:15:03.403966 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.403948 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" Sep 14 05:15:03.404070 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.403964 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw" event={"ID":"adb3e55f-0c5a-4119-8c4a-74c052efb949","Type":"ContainerDied","Data":"d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205"} Sep 14 05:15:03.404070 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.404000 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d26a48e8f767595fc023ba21c43fc91bd3a416afc46a2972a145461de53d9205" Sep 14 05:15:03.433141 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.433105 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:03.435041 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.435021 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2eeb50-29822715-d65nw"] Sep 14 05:15:03.552513 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.552483 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395552f4-00ef-40d6-be81-e1ece10d310c" path="/var/lib/kubelet/pods/395552f4-00ef-40d6-be81-e1ece10d310c/volumes" Sep 14 05:15:03.552865 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.552852 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" path="/var/lib/kubelet/pods/3f0cc9c6-00d4-4b17-b212-5e8aac5eda36/volumes" Sep 14 05:15:03.553129 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:03.553106 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb3e55f-0c5a-4119-8c4a-74c052efb949" path="/var/lib/kubelet/pods/adb3e55f-0c5a-4119-8c4a-74c052efb949/volumes" Sep 14 05:15:09.171180 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:09.171137 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:15:09.171687 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:09.171400 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" containerID="cri-o://d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b" gracePeriod=30 Sep 14 05:15:13.144846 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:13.144805 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 05:15:13.145356 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:13.144875 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 05:15:14.437136 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.437057 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:15:14.468414 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.468384 2581 generic.go:356] "Generic (PLEG): container finished" podID="48713d24-602b-4485-9f76-5466dba96659" containerID="d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b" exitCode=0 Sep 14 05:15:14.468564 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.468429 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" Sep 14 05:15:14.468564 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.468462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerDied","Data":"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b"} Sep 14 05:15:14.468564 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.468520 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb" event={"ID":"48713d24-602b-4485-9f76-5466dba96659","Type":"ContainerDied","Data":"cac6ee92f636bfba59f26a8b9782693097d4eaf6598536893ced49f0d3b473c9"} Sep 14 05:15:14.468564 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.468536 2581 scope.go:117] "RemoveContainer" containerID="d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b" Sep 14 05:15:14.477623 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.477607 2581 scope.go:117] "RemoveContainer" containerID="92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b" Sep 14 05:15:14.485822 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.485806 2581 scope.go:117] "RemoveContainer" containerID="d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b" Sep 14 05:15:14.486087 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:15:14.486068 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b\": container with ID starting with d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b not found: ID does not exist" containerID="d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b" Sep 14 05:15:14.486187 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.486095 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b"} err="failed to get container status \"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b\": rpc error: code = NotFound desc = could not find container \"d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b\": container with ID starting with d26ab4b7baf0d320e6cbfa4035f2a6574e041d300833d2e5fda0f8c1ddd4c27b not found: ID does not exist" Sep 14 05:15:14.486187 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.486138 2581 scope.go:117] "RemoveContainer" containerID="92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b" Sep 14 05:15:14.486391 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:15:14.486373 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b\": container with ID starting with 92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b not found: ID does not exist" containerID="92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b" Sep 14 05:15:14.486449 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.486398 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b"} err="failed to get container status \"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b\": rpc error: code = NotFound desc = could not find container \"92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b\": container with ID starting with 92c23c2b7a80a78459fa990b10b743bfb1b962d51893364f83fcf632e1615d0b not found: ID does not exist" Sep 14 05:15:14.547142 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547101 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547151 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547207 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547236 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547273 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpktc\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547294 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547350 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547406 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:14.547557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547452 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547551 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket\") pod \"48713d24-602b-4485-9f76-5466dba96659\" (UID: \"48713d24-602b-4485-9f76-5466dba96659\") " Sep 14 05:15:14.547822 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547587 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data" (OuterVolumeSpecName: "istio-data") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:14.547822 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:14.547931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547837 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:14.547931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547850 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.547931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547868 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.547931 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547877 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.548151 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.547947 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:15:14.549480 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.549453 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:15:14.549480 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.549470 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc" (OuterVolumeSpecName: "kube-api-access-gpktc") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "kube-api-access-gpktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:15:14.549615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.549462 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token" (OuterVolumeSpecName: "istio-token") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:15:14.549669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.549643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48713d24-602b-4485-9f76-5466dba96659" (UID: "48713d24-602b-4485-9f76-5466dba96659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:14.649500 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649469 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.649500 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649500 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48713d24-602b-4485-9f76-5466dba96659-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.649670 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649514 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpktc\" (UniqueName: \"kubernetes.io/projected/48713d24-602b-4485-9f76-5466dba96659-kube-api-access-gpktc\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.649670 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649530 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48713d24-602b-4485-9f76-5466dba96659-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.649670 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649545 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.649670 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.649559 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48713d24-602b-4485-9f76-5466dba96659-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:14.790272 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.790243 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:15:14.795618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:14.795586 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2eeb50-openshift-default-85f875f444-mhrvb"] Sep 14 05:15:15.552852 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:15.552816 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48713d24-602b-4485-9f76-5466dba96659" path="/var/lib/kubelet/pods/48713d24-602b-4485-9f76-5466dba96659/volumes" Sep 14 05:15:26.777791 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:26.777754 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:15:26.778505 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:26.778301 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" containerID="cri-o://ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e" gracePeriod=30 Sep 14 05:15:32.032169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.032140 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:15:32.111304 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111276 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111456 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111340 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111456 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111356 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111545 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111517 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111595 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111571 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111655 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111610 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:32.111655 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111625 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:15:32.111655 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111633 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111669 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111696 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111724 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtw2r\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r\") pod \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\" (UID: \"f0a115b4-27de-4cf3-8ed7-3b9383d03463\") " Sep 14 05:15:32.111816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111808 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:32.111994 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111807 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:32.111994 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.111916 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:32.112205 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.112179 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.112205 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.112206 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.112415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.112220 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.112415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.112235 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.112415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.112250 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.113631 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.113603 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:15:32.113772 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.113749 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:15:32.113820 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.113777 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:15:32.113927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.113913 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r" (OuterVolumeSpecName: "kube-api-access-xtw2r") pod "f0a115b4-27de-4cf3-8ed7-3b9383d03463" (UID: "f0a115b4-27de-4cf3-8ed7-3b9383d03463"). InnerVolumeSpecName "kube-api-access-xtw2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:15:32.213096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.213064 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.213096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.213090 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.213096 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.213099 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtw2r\" (UniqueName: \"kubernetes.io/projected/f0a115b4-27de-4cf3-8ed7-3b9383d03463-kube-api-access-xtw2r\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.213317 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.213109 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a115b4-27de-4cf3-8ed7-3b9383d03463-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:15:32.572510 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.572426 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerID="ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e" exitCode=0 Sep 14 05:15:32.572510 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.572479 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" Sep 14 05:15:32.572695 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.572513 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerDied","Data":"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e"} Sep 14 05:15:32.572695 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.572574 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64" event={"ID":"f0a115b4-27de-4cf3-8ed7-3b9383d03463","Type":"ContainerDied","Data":"6defe7ef19566d74ace45036f25854fd48dde74e3b69f0b0fed8e9331c6bd708"} Sep 14 05:15:32.572695 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.572605 2581 scope.go:117] "RemoveContainer" containerID="ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e" Sep 14 05:15:32.586654 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.586627 2581 scope.go:117] "RemoveContainer" containerID="520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa" Sep 14 05:15:32.596641 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.596622 2581 scope.go:117] "RemoveContainer" containerID="ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e" Sep 14 05:15:32.596906 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:15:32.596889 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e\": container with ID starting with ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e not found: ID does not exist" containerID="ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e" Sep 14 05:15:32.597007 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.596914 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e"} err="failed to get container status \"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e\": rpc error: code = NotFound desc = could not find container \"ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e\": container with ID starting with ba63fac8633a11c27ca77ef62edd5c7b2cae569b8c4a9d8a573c10d3fc075f0e not found: ID does not exist" Sep 14 05:15:32.597007 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.596931 2581 scope.go:117] "RemoveContainer" containerID="520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa" Sep 14 05:15:32.597182 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.597153 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:15:32.597278 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:15:32.597187 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa\": container with ID starting with 520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa not found: ID does not exist" containerID="520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa" Sep 14 05:15:32.597278 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.597208 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa"} err="failed to get container status \"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa\": rpc error: code = NotFound desc = could not find container \"520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa\": container with ID starting with 520e7dca2758cebeff23657570fe3fd83e044d275b4934874b6bb10118a0bafa not found: ID does not exist" Sep 14 05:15:32.601258 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:32.601236 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a8c46f-openshift-default-5c9bd8464d-8vl64"] Sep 14 05:15:33.554322 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:15:33.554285 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" path="/var/lib/kubelet/pods/f0a115b4-27de-4cf3-8ed7-3b9383d03463/volumes" Sep 14 05:16:29.853178 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:16:29.853144 2581 scope.go:117] "RemoveContainer" containerID="b248c45d3f1054f86bf00f06b510e43b56d2788cca354a5bc5862a966d66cb22" Sep 14 05:17:24.193417 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193385 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193904 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31e50f0-7922-481a-b8e6-324411da33a2" containerName="cleanup" Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193922 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31e50f0-7922-481a-b8e6-324411da33a2" containerName="cleanup" Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193939 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193947 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193969 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:24.193973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.193977 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194001 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb3e55f-0c5a-4119-8c4a-74c052efb949" containerName="cleanup" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194009 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb3e55f-0c5a-4119-8c4a-74c052efb949" containerName="cleanup" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194038 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" containerName="payload-processing" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194046 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" containerName="payload-processing" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194056 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194064 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194187 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31e50f0-7922-481a-b8e6-324411da33a2" containerName="cleanup" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194200 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194213 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194226 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb3e55f-0c5a-4119-8c4a-74c052efb949" containerName="cleanup" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194238 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0cc9c6-00d4-4b17-b212-5e8aac5eda36" containerName="payload-processing" Sep 14 05:17:24.194285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.194248 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:24.197744 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.197717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.199652 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.199631 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-524a38-openshift-default-dockercfg-lsdjk\"" Sep 14 05:17:24.209668 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.209633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:17:24.269004 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.268956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h226c\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.269445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.269305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370442 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h226c\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.370656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370939 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.370978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371344 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.371037 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371344 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.371164 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.371511 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.371491 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.372969 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.372950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.373217 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.373198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.377470 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.377441 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.377666 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.377650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h226c\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c\") pod \"e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.511391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.511308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:24.789212 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.789185 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:17:24.791282 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:17:24.791257 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95caa4af_941a_4572_8c12_010ea6201a17.slice/crio-58d7644b388da9d54e960499e612a9c50bcb1805f638313e739716e070774e22 WatchSource:0}: Error finding container 58d7644b388da9d54e960499e612a9c50bcb1805f638313e739716e070774e22: Status 404 returned error can't find the container with id 58d7644b388da9d54e960499e612a9c50bcb1805f638313e739716e070774e22 Sep 14 05:17:24.793606 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.793574 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:24.793702 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.793639 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:24.793702 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:24.793667 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:25.168926 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:25.168885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerStarted","Data":"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef"} Sep 14 05:17:25.169109 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:25.168951 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerStarted","Data":"58d7644b388da9d54e960499e612a9c50bcb1805f638313e739716e070774e22"} Sep 14 05:17:25.188364 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:25.188314 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podStartSLOduration=1.188298562 podStartE2EDuration="1.188298562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:17:25.185700352 +0000 UTC m=+1856.143317088" watchObservedRunningTime="2026-09-14 05:17:25.188298562 +0000 UTC m=+1856.145915299" Sep 14 05:17:25.512178 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:25.512077 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:25.517219 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:25.517191 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:26.175362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:26.175333 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:26.176393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:26.176374 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:29.521588 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.521555 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:17:29.521976 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.521963 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:29.522027 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.521978 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48713d24-602b-4485-9f76-5466dba96659" containerName="istio-proxy" Sep 14 05:17:29.522065 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.522058 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a115b4-27de-4cf3-8ed7-3b9383d03463" containerName="istio-proxy" Sep 14 05:17:29.525237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.525222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.527164 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.527138 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-524a38\"" Sep 14 05:17:29.527296 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.527231 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-524a38-dockercfg-8qzn9\"" Sep 14 05:17:29.535248 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.535222 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:17:29.622075 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.622036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.622277 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.622091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvj2\" (UniqueName: \"kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.723694 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.723655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.723898 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.723719 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvj2\" (UniqueName: \"kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.724523 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.724498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.731237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.731211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvj2\" (UniqueName: \"kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2\") pod \"payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:29.838298 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:29.838208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:30.130128 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:30.130092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:17:30.199816 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:30.199788 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" event={"ID":"94814009-1ed7-40c5-b5e3-01ae4088bb59","Type":"ContainerStarted","Data":"f282412d5051d302d2e118388fd61ce0ad1559673d7a294856bf7f19d10ee832"} Sep 14 05:17:31.206452 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:31.206417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" event={"ID":"94814009-1ed7-40c5-b5e3-01ae4088bb59","Type":"ContainerStarted","Data":"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f"} Sep 14 05:17:31.206837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:31.206496 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:31.225136 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:31.222880 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" podStartSLOduration=2.222860227 podStartE2EDuration="2.222860227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:17:31.220439337 +0000 UTC m=+1862.178056073" watchObservedRunningTime="2026-09-14 05:17:31.222860227 +0000 UTC m=+1862.180476965" Sep 14 05:17:42.214628 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:42.214593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:17:42.217252 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:42.217223 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 05:17:42.217380 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:42.217273 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 05:17:43.282809 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.282776 2581 generic.go:356] "Generic (PLEG): container finished" podID="95caa4af-941a-4572-8c12-010ea6201a17" containerID="d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef" exitCode=0 Sep 14 05:17:43.283166 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.282849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerDied","Data":"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef"} Sep 14 05:17:43.283304 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.283289 2581 scope.go:117] "RemoveContainer" containerID="d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef" Sep 14 05:17:43.284379 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.284348 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:43.284465 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.284416 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:43.284465 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:43.284447 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:17:44.290378 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:44.290341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerStarted","Data":"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1"} Sep 14 05:17:44.512268 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:44.512230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:44.512557 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:44.512531 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 14 05:17:44.512622 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:44.512592 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 14 05:17:45.512498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:45.512464 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 14 05:17:45.512872 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:45.512519 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 14 05:17:46.511751 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:46.511712 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 14 05:17:46.511938 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:46.511768 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 14 05:17:47.511702 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:47.511660 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 14 05:17:47.512194 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:47.511717 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 14 05:17:48.512765 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:48.512721 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 14 05:17:48.513213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:48.512785 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 14 05:17:49.522763 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:49.522728 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:49.524381 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:49.524347 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:49.525905 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:49.525883 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:17:59.978582 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:59.978479 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:17:59.987740 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:59.987673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:17:59.990393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:59.990183 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7bbea9\"" Sep 14 05:17:59.990570 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:59.990503 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7bbea9-dockercfg-k9skg\"" Sep 14 05:17:59.995756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:17:59.995290 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:18:00.022079 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.022035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.022274 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.022196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdk4\" (UniqueName: \"kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.122749 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.122692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.122947 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.122860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdk4\" (UniqueName: \"kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.123496 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.123478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.131149 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.131094 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdk4\" (UniqueName: \"kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4\") pod \"payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.304275 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.304174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:00.577794 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:00.577766 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:18:00.579890 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:18:00.579858 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddde9f71_72ca_4290_98be_4001f8a560dc.slice/crio-6816416f537f32a2884d019d24dc6a5033d1698f4e94aa361baebaa097ab548a WatchSource:0}: Error finding container 6816416f537f32a2884d019d24dc6a5033d1698f4e94aa361baebaa097ab548a: Status 404 returned error can't find the container with id 6816416f537f32a2884d019d24dc6a5033d1698f4e94aa361baebaa097ab548a Sep 14 05:18:01.388708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:01.388670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" event={"ID":"ddde9f71-72ca-4290-98be-4001f8a560dc","Type":"ContainerStarted","Data":"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b"} Sep 14 05:18:01.389223 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:01.388729 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" event={"ID":"ddde9f71-72ca-4290-98be-4001f8a560dc","Type":"ContainerStarted","Data":"6816416f537f32a2884d019d24dc6a5033d1698f4e94aa361baebaa097ab548a"} Sep 14 05:18:01.389223 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:01.388813 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:01.403586 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:01.403515 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" podStartSLOduration=2.403490983 podStartE2EDuration="2.403490983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:18:01.402212259 +0000 UTC m=+1892.359828996" watchObservedRunningTime="2026-09-14 05:18:01.403490983 +0000 UTC m=+1892.361107721" Sep 14 05:18:12.397341 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:12.397300 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:18:27.550490 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:27.550456 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:18:27.673386 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:18:27.673361 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf_ddde9f71-72ca-4290-98be-4001f8a560dc/payload-processing/0.log" Sep 14 05:19:08.932305 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:08.932264 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:19:09.054811 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.054779 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf_ddde9f71-72ca-4290-98be-4001f8a560dc/payload-processing/0.log" Sep 14 05:19:09.665565 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.665533 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:19:09.669242 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.669221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.671265 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.671244 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2339fb-openshift-default-dockercfg-cllmd\"" Sep 14 05:19:09.681612 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.681591 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:19:09.813613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ppk\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813813 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813813 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813813 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813923 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813923 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.813923 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.814021 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.813925 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.914703 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.914864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.914864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.914864 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.914977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915250 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ppk\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915361 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915438 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915415 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915481 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915426 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915481 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.915735 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.915662 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.917310 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.917291 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.917415 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.917294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.922782 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.922759 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ppk\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.922873 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.922855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token\") pod \"e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:09.979658 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:09.979640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:10.260554 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.260527 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:19:10.262286 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:19:10.262245 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedeede7_970c_4ea0_b7ee_afc7adf6d18d.slice/crio-07be4795010824d7eec6ed01cc0aa174f8f47f32f561505ef3c59812abb636f8 WatchSource:0}: Error finding container 07be4795010824d7eec6ed01cc0aa174f8f47f32f561505ef3c59812abb636f8: Status 404 returned error can't find the container with id 07be4795010824d7eec6ed01cc0aa174f8f47f32f561505ef3c59812abb636f8 Sep 14 05:19:10.264487 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.264454 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:10.264565 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.264522 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:10.264565 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.264550 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:10.770238 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.770187 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerStarted","Data":"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0"} Sep 14 05:19:10.770405 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.770264 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerStarted","Data":"07be4795010824d7eec6ed01cc0aa174f8f47f32f561505ef3c59812abb636f8"} Sep 14 05:19:10.792924 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.792879 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podStartSLOduration=1.792864671 podStartE2EDuration="1.792864671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:19:10.790577023 +0000 UTC m=+1961.748193759" watchObservedRunningTime="2026-09-14 05:19:10.792864671 +0000 UTC m=+1961.750481407" Sep 14 05:19:10.980172 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:10.980141 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:11.985337 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:11.985304 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:12.784282 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:12.784249 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:12.785237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:12.785220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:29.897837 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:29.897802 2581 scope.go:117] "RemoveContainer" containerID="4786e976f0c92e40e2e4baa05bd1adfeb8e17a17c245cf1d5ceabf6e4a54a5bb" Sep 14 05:19:32.903619 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.903564 2581 generic.go:356] "Generic (PLEG): container finished" podID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerID="187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0" exitCode=0 Sep 14 05:19:32.904168 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.903635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerDied","Data":"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0"} Sep 14 05:19:32.904168 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.904016 2581 scope.go:117] "RemoveContainer" containerID="187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0" Sep 14 05:19:32.905209 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.905183 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:32.905282 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.905255 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:32.905321 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:32.905287 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 14 05:19:33.910566 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:33.910532 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerStarted","Data":"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833"} Sep 14 05:19:33.979881 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:33.979846 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:33.980073 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:33.980049 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:33.980159 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:33.980139 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:34.980443 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:34.980402 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:34.980811 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:34.980471 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:35.980429 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:35.980393 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:35.980613 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:35.980459 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:36.980237 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:36.980202 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:36.980451 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:36.980256 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:37.981008 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:37.980960 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:37.981503 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:37.981042 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:38.980000 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:38.979967 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 14 05:19:38.980220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:38.980055 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 14 05:19:39.984619 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:39.984582 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:39.985014 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:39.984984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:39.985845 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:39.985821 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:19:43.714204 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.714166 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj"] Sep 14 05:19:43.718210 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.718186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:43.720489 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.720462 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:19:43.720615 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.720529 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:19:43.730441 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.730412 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj"] Sep 14 05:19:43.769441 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.769406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj\" (UID: \"88aa4852-03cc-4206-94f8-8bdc41863a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:43.869949 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.869908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj\" (UID: \"88aa4852-03cc-4206-94f8-8bdc41863a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:43.870618 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:43.870595 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj\" (UID: \"88aa4852-03cc-4206-94f8-8bdc41863a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:44.029247 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:44.029163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:44.303014 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:44.302987 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj"] Sep 14 05:19:44.305174 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:19:44.305147 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aa4852_03cc_4206_94f8_8bdc41863a3e.slice/crio-68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755 WatchSource:0}: Error finding container 68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755: Status 404 returned error can't find the container with id 68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755 Sep 14 05:19:44.972423 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:44.972386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" event={"ID":"88aa4852-03cc-4206-94f8-8bdc41863a3e","Type":"ContainerStarted","Data":"54b1ca8989bb63f3c93577304139a65cabddc9b2d879433de844fbda7ae6a89d"} Sep 14 05:19:44.972796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:44.972440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" event={"ID":"88aa4852-03cc-4206-94f8-8bdc41863a3e","Type":"ContainerStarted","Data":"68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755"} Sep 14 05:19:44.995243 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:44.995201 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" podStartSLOduration=1.995187788 podStartE2EDuration="1.995187788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:19:44.992324239 +0000 UTC m=+1995.949940989" watchObservedRunningTime="2026-09-14 05:19:44.995187788 +0000 UTC m=+1995.952804524" Sep 14 05:19:45.979419 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:45.979383 2581 generic.go:356] "Generic (PLEG): container finished" podID="88aa4852-03cc-4206-94f8-8bdc41863a3e" containerID="54b1ca8989bb63f3c93577304139a65cabddc9b2d879433de844fbda7ae6a89d" exitCode=0 Sep 14 05:19:45.979818 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:45.979474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" event={"ID":"88aa4852-03cc-4206-94f8-8bdc41863a3e","Type":"ContainerDied","Data":"54b1ca8989bb63f3c93577304139a65cabddc9b2d879433de844fbda7ae6a89d"} Sep 14 05:19:47.125792 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.125761 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:47.202358 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.202321 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca\") pod \"88aa4852-03cc-4206-94f8-8bdc41863a3e\" (UID: \"88aa4852-03cc-4206-94f8-8bdc41863a3e\") " Sep 14 05:19:47.202693 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.202671 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88aa4852-03cc-4206-94f8-8bdc41863a3e" (UID: "88aa4852-03cc-4206-94f8-8bdc41863a3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:19:47.304233 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.304155 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88aa4852-03cc-4206-94f8-8bdc41863a3e-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:19:47.991730 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.991701 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" Sep 14 05:19:47.991899 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.991734 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj" event={"ID":"88aa4852-03cc-4206-94f8-8bdc41863a3e","Type":"ContainerDied","Data":"68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755"} Sep 14 05:19:47.991899 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:47.991782 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68655a1d83fc5be874601d2b3cb18933cd576434dd97e6841329133fd8eca755" Sep 14 05:19:54.869436 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:54.869383 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj"] Sep 14 05:19:54.871651 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:54.871625 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2339fb-4n2cj"] Sep 14 05:19:55.553533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:19:55.553499 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88aa4852-03cc-4206-94f8-8bdc41863a3e" path="/var/lib/kubelet/pods/88aa4852-03cc-4206-94f8-8bdc41863a3e/volumes" Sep 14 05:20:01.388190 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:01.388157 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:20:01.388766 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:01.388377 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" containerID="cri-o://8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833" gracePeriod=30 Sep 14 05:20:06.653811 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.653746 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:20:06.799962 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.799926 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800019 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800069 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800088 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800112 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800147 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800429 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800167 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6ppk\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800429 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800216 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800429 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800241 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token\") pod \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\" (UID: \"cedeede7-970c-4ea0-b7ee-afc7adf6d18d\") " Sep 14 05:20:06.800429 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800355 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:20:06.800649 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800438 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:20:06.800649 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800505 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:20:06.800767 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800654 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:20:06.800767 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800730 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:20:06.800877 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800811 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.800877 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800831 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.800877 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800844 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.800877 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.800859 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.802357 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.802331 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:20:06.802477 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.802370 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:20:06.802477 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.802459 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:20:06.802603 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.802561 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk" (OuterVolumeSpecName: "kube-api-access-z6ppk") pod "cedeede7-970c-4ea0-b7ee-afc7adf6d18d" (UID: "cedeede7-970c-4ea0-b7ee-afc7adf6d18d"). InnerVolumeSpecName "kube-api-access-z6ppk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:20:06.902466 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.902435 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.902466 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.902466 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.902634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.902476 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.902634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.902486 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ppk\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-kube-api-access-z6ppk\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:06.902634 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:06.902497 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedeede7-970c-4ea0-b7ee-afc7adf6d18d-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:07.115556 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.115516 2581 generic.go:356] "Generic (PLEG): container finished" podID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerID="8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833" exitCode=0 Sep 14 05:20:07.115796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.115571 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" Sep 14 05:20:07.115796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.115607 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerDied","Data":"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833"} Sep 14 05:20:07.115796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.115664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld" event={"ID":"cedeede7-970c-4ea0-b7ee-afc7adf6d18d","Type":"ContainerDied","Data":"07be4795010824d7eec6ed01cc0aa174f8f47f32f561505ef3c59812abb636f8"} Sep 14 05:20:07.115796 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.115687 2581 scope.go:117] "RemoveContainer" containerID="8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833" Sep 14 05:20:07.125806 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.125782 2581 scope.go:117] "RemoveContainer" containerID="187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0" Sep 14 05:20:07.134902 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.134881 2581 scope.go:117] "RemoveContainer" containerID="8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833" Sep 14 05:20:07.135209 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:20:07.135189 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833\": container with ID starting with 8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833 not found: ID does not exist" containerID="8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833" Sep 14 05:20:07.135300 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.135222 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833"} err="failed to get container status \"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833\": rpc error: code = NotFound desc = could not find container \"8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833\": container with ID starting with 8300148b33377328c0053fbeba2f321f925cdb224092ebf1448aa0afd27bf833 not found: ID does not exist" Sep 14 05:20:07.135300 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.135246 2581 scope.go:117] "RemoveContainer" containerID="187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0" Sep 14 05:20:07.135521 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:20:07.135504 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0\": container with ID starting with 187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0 not found: ID does not exist" containerID="187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0" Sep 14 05:20:07.135561 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.135527 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0"} err="failed to get container status \"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0\": rpc error: code = NotFound desc = could not find container \"187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0\": container with ID starting with 187a883c616707a7660a2d399d67ac8ca8bc3eeee5fa7a2b3c1b64055dd56ba0 not found: ID does not exist" Sep 14 05:20:07.138909 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.138882 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:20:07.146568 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.146544 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2339fb-openshift-default-7cb66954b6-8szld"] Sep 14 05:20:07.552979 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:07.552893 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" path="/var/lib/kubelet/pods/cedeede7-970c-4ea0-b7ee-afc7adf6d18d/volumes" Sep 14 05:20:21.096464 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.096427 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:20:21.097068 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.096651 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" podUID="94814009-1ed7-40c5-b5e3-01ae4088bb59" containerName="payload-pre-processing" containerID="cri-o://13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f" gracePeriod=30 Sep 14 05:20:21.371208 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.371170 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:20:21.451042 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.450991 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume\") pod \"94814009-1ed7-40c5-b5e3-01ae4088bb59\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " Sep 14 05:20:21.451335 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.451107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvj2\" (UniqueName: \"kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2\") pod \"94814009-1ed7-40c5-b5e3-01ae4088bb59\" (UID: \"94814009-1ed7-40c5-b5e3-01ae4088bb59\") " Sep 14 05:20:21.451603 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.451559 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94814009-1ed7-40c5-b5e3-01ae4088bb59" (UID: "94814009-1ed7-40c5-b5e3-01ae4088bb59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:20:21.453732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.453696 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2" (OuterVolumeSpecName: "kube-api-access-nvvj2") pod "94814009-1ed7-40c5-b5e3-01ae4088bb59" (UID: "94814009-1ed7-40c5-b5e3-01ae4088bb59"). InnerVolumeSpecName "kube-api-access-nvvj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:20:21.552713 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.552681 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvj2\" (UniqueName: \"kubernetes.io/projected/94814009-1ed7-40c5-b5e3-01ae4088bb59-kube-api-access-nvvj2\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:21.552713 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:21.552708 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94814009-1ed7-40c5-b5e3-01ae4088bb59-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:20:22.204208 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.204168 2581 generic.go:356] "Generic (PLEG): container finished" podID="94814009-1ed7-40c5-b5e3-01ae4088bb59" containerID="13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f" exitCode=0 Sep 14 05:20:22.204708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.204220 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" Sep 14 05:20:22.204708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.204260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" event={"ID":"94814009-1ed7-40c5-b5e3-01ae4088bb59","Type":"ContainerDied","Data":"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f"} Sep 14 05:20:22.204708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.204318 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd" event={"ID":"94814009-1ed7-40c5-b5e3-01ae4088bb59","Type":"ContainerDied","Data":"f282412d5051d302d2e118388fd61ce0ad1559673d7a294856bf7f19d10ee832"} Sep 14 05:20:22.204708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.204344 2581 scope.go:117] "RemoveContainer" containerID="13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f" Sep 14 05:20:22.213777 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.213754 2581 scope.go:117] "RemoveContainer" containerID="13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f" Sep 14 05:20:22.214142 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:20:22.214106 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f\": container with ID starting with 13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f not found: ID does not exist" containerID="13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f" Sep 14 05:20:22.214228 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.214151 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f"} err="failed to get container status \"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f\": rpc error: code = NotFound desc = could not find container \"13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f\": container with ID starting with 13cd70a3b863ca7c19d554855ec4ef1a4b7ab0f95ce2de963f543258f579de8f not found: ID does not exist" Sep 14 05:20:22.220421 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.220394 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:20:22.222465 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:22.222442 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-524a38-86b8879754-kkckd"] Sep 14 05:20:23.554016 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:20:23.553983 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94814009-1ed7-40c5-b5e3-01ae4088bb59" path="/var/lib/kubelet/pods/94814009-1ed7-40c5-b5e3-01ae4088bb59/volumes" Sep 14 05:21:10.938956 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:10.938921 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:21:10.939391 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:10.939105 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" containerID="cri-o://20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1" gracePeriod=30 Sep 14 05:21:16.194776 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.194749 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:21:16.281784 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281680 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.281784 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281742 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.281784 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281763 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281834 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h226c\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281871 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281907 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281930 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.281982 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data" (OuterVolumeSpecName: "istio-data") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282022 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token\") pod \"95caa4af-941a-4572-8c12-010ea6201a17\" (UID: \"95caa4af-941a-4572-8c12-010ea6201a17\") " Sep 14 05:21:16.282101 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282034 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282128 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282263 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282470 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-data\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282489 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-certs\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282492 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282502 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-workload-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.282571 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.282547 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-credential-socket\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.284090 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.284061 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 05:21:16.284411 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.284387 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token" (OuterVolumeSpecName: "istio-token") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:16.284541 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.284482 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c" (OuterVolumeSpecName: "kube-api-access-h226c") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "kube-api-access-h226c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:16.284541 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.284503 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95caa4af-941a-4572-8c12-010ea6201a17" (UID: "95caa4af-941a-4572-8c12-010ea6201a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 05:21:16.383439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.383389 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95caa4af-941a-4572-8c12-010ea6201a17-istio-podinfo\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.383439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.383434 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-istio-token\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.383439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.383445 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95caa4af-941a-4572-8c12-010ea6201a17-istio-envoy\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.383439 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.383454 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h226c\" (UniqueName: \"kubernetes.io/projected/95caa4af-941a-4572-8c12-010ea6201a17-kube-api-access-h226c\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.383723 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.383465 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95caa4af-941a-4572-8c12-010ea6201a17-istiod-ca-cert\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:16.497184 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.497150 2581 generic.go:356] "Generic (PLEG): container finished" podID="95caa4af-941a-4572-8c12-010ea6201a17" containerID="20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1" exitCode=0 Sep 14 05:21:16.497353 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.497239 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" Sep 14 05:21:16.497353 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.497267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerDied","Data":"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1"} Sep 14 05:21:16.497353 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.497313 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm" event={"ID":"95caa4af-941a-4572-8c12-010ea6201a17","Type":"ContainerDied","Data":"58d7644b388da9d54e960499e612a9c50bcb1805f638313e739716e070774e22"} Sep 14 05:21:16.497353 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.497338 2581 scope.go:117] "RemoveContainer" containerID="20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1" Sep 14 05:21:16.507169 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.507149 2581 scope.go:117] "RemoveContainer" containerID="d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef" Sep 14 05:21:16.515872 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.515852 2581 scope.go:117] "RemoveContainer" containerID="20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1" Sep 14 05:21:16.516281 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:16.516255 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1\": container with ID starting with 20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1 not found: ID does not exist" containerID="20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1" Sep 14 05:21:16.516403 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.516289 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1"} err="failed to get container status \"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1\": rpc error: code = NotFound desc = could not find container \"20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1\": container with ID starting with 20f9a33db5dac980c4d4971620b74317ee4d68a6d31fc30ec00d4ac7e8ade3d1 not found: ID does not exist" Sep 14 05:21:16.516403 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.516315 2581 scope.go:117] "RemoveContainer" containerID="d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef" Sep 14 05:21:16.516646 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:16.516619 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef\": container with ID starting with d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef not found: ID does not exist" containerID="d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef" Sep 14 05:21:16.516731 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.516673 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef"} err="failed to get container status \"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef\": rpc error: code = NotFound desc = could not find container \"d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef\": container with ID starting with d63ddca8062a51d3427f9c90ee883f7cf979274834b6f337ac9856aae0f13fef not found: ID does not exist" Sep 14 05:21:16.519317 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.519295 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:21:16.523293 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:16.523271 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-524a38-openshift-default-d897559fb-97xvm"] Sep 14 05:21:17.552476 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:17.552445 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95caa4af-941a-4572-8c12-010ea6201a17" path="/var/lib/kubelet/pods/95caa4af-941a-4572-8c12-010ea6201a17/volumes" Sep 14 05:21:22.155750 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.155714 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:21:22.156756 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.155988 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerName="payload-processing" containerID="cri-o://e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b" gracePeriod=30 Sep 14 05:21:22.418357 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.418283 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:21:22.531271 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.531236 2581 generic.go:356] "Generic (PLEG): container finished" podID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerID="e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b" exitCode=0 Sep 14 05:21:22.531434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.531274 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" Sep 14 05:21:22.531434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.531319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" event={"ID":"ddde9f71-72ca-4290-98be-4001f8a560dc","Type":"ContainerDied","Data":"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b"} Sep 14 05:21:22.531434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.531370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" event={"ID":"ddde9f71-72ca-4290-98be-4001f8a560dc","Type":"ContainerDied","Data":"6816416f537f32a2884d019d24dc6a5033d1698f4e94aa361baebaa097ab548a"} Sep 14 05:21:22.531434 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.531394 2581 scope.go:117] "RemoveContainer" containerID="e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b" Sep 14 05:21:22.539955 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.539938 2581 scope.go:117] "RemoveContainer" containerID="e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b" Sep 14 05:21:22.540256 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:22.540222 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b\": container with ID starting with e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b not found: ID does not exist" containerID="e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b" Sep 14 05:21:22.540362 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.540260 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b"} err="failed to get container status \"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b\": rpc error: code = NotFound desc = could not find container \"e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b\": container with ID starting with e83d385745864299fb0383a96d2874708aead452795aa8bfe3a980e7ad76d01b not found: ID does not exist" Sep 14 05:21:22.549728 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.549708 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rdk4\" (UniqueName: \"kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4\") pod \"ddde9f71-72ca-4290-98be-4001f8a560dc\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " Sep 14 05:21:22.549832 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.549749 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume\") pod \"ddde9f71-72ca-4290-98be-4001f8a560dc\" (UID: \"ddde9f71-72ca-4290-98be-4001f8a560dc\") " Sep 14 05:21:22.550186 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.550167 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddde9f71-72ca-4290-98be-4001f8a560dc" (UID: "ddde9f71-72ca-4290-98be-4001f8a560dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:22.551743 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.551726 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4" (OuterVolumeSpecName: "kube-api-access-8rdk4") pod "ddde9f71-72ca-4290-98be-4001f8a560dc" (UID: "ddde9f71-72ca-4290-98be-4001f8a560dc"). InnerVolumeSpecName "kube-api-access-8rdk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:22.651402 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.651378 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdk4\" (UniqueName: \"kubernetes.io/projected/ddde9f71-72ca-4290-98be-4001f8a560dc-kube-api-access-8rdk4\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:22.651402 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.651400 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddde9f71-72ca-4290-98be-4001f8a560dc-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:22.853510 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.853480 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:21:22.855402 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:22.855380 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf"] Sep 14 05:21:23.395502 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:23.395461 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-7bbea9-95557b9f-hkvwf" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.73:9004: i/o timeout" Sep 14 05:21:23.552526 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:23.552492 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" path="/var/lib/kubelet/pods/ddde9f71-72ca-4290-98be-4001f8a560dc/volumes" Sep 14 05:21:29.552715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.552686 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj"] Sep 14 05:21:29.553075 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553055 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94814009-1ed7-40c5-b5e3-01ae4088bb59" containerName="payload-pre-processing" Sep 14 05:21:29.553075 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553066 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94814009-1ed7-40c5-b5e3-01ae4088bb59" containerName="payload-pre-processing" Sep 14 05:21:29.553075 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553074 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerName="payload-processing" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553079 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerName="payload-processing" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553088 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88aa4852-03cc-4206-94f8-8bdc41863a3e" containerName="revoke-keys" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553094 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88aa4852-03cc-4206-94f8-8bdc41863a3e" containerName="revoke-keys" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553104 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553109 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553140 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553146 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553151 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553156 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553172 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.553213 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553177 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553239 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddde9f71-72ca-4290-98be-4001f8a560dc" containerName="payload-processing" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553248 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553253 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553258 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cedeede7-970c-4ea0-b7ee-afc7adf6d18d" containerName="istio-proxy" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553264 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88aa4852-03cc-4206-94f8-8bdc41863a3e" containerName="revoke-keys" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553269 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94814009-1ed7-40c5-b5e3-01ae4088bb59" containerName="payload-pre-processing" Sep 14 05:21:29.553539 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.553276 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95caa4af-941a-4572-8c12-010ea6201a17" containerName="istio-proxy" Sep 14 05:21:29.557804 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.557779 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:29.558252 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.558226 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj"] Sep 14 05:21:29.559854 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.559836 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:21:29.559950 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.559851 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:21:29.714705 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.714673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj\" (UID: \"96c15dbf-c175-42ce-a0f1-3d74224b66ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:29.816162 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.816053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj\" (UID: \"96c15dbf-c175-42ce-a0f1-3d74224b66ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:29.816684 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.816663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj\" (UID: \"96c15dbf-c175-42ce-a0f1-3d74224b66ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:29.869602 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.869570 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:29.938336 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.937982 2581 scope.go:117] "RemoveContainer" containerID="6043c9d2bd0549633e63d6887af7fbc2ba381781eae541a70837f7c438656242" Sep 14 05:21:29.948467 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:29.948438 2581 scope.go:117] "RemoveContainer" containerID="8a55a8b177e87428afa22785698f4c6184a153f08ef5a873a1b7213872f6a35f" Sep 14 05:21:30.142057 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:30.142032 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj"] Sep 14 05:21:30.144452 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:21:30.144426 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c15dbf_c175_42ce_a0f1_3d74224b66ba.slice/crio-9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0 WatchSource:0}: Error finding container 9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0: Status 404 returned error can't find the container with id 9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0 Sep 14 05:21:30.576081 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:30.576046 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" event={"ID":"96c15dbf-c175-42ce-a0f1-3d74224b66ba","Type":"ContainerStarted","Data":"5ed44cb73485876dc07ca5f65f00efe8b6d26455193ec7a93bdf67aac977fd17"} Sep 14 05:21:30.576484 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:30.576095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" event={"ID":"96c15dbf-c175-42ce-a0f1-3d74224b66ba","Type":"ContainerStarted","Data":"9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0"} Sep 14 05:21:30.592840 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:30.592781 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" podStartSLOduration=1.59276412 podStartE2EDuration="1.59276412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:21:30.591859613 +0000 UTC m=+2101.549476350" watchObservedRunningTime="2026-09-14 05:21:30.59276412 +0000 UTC m=+2101.550380858" Sep 14 05:21:31.582836 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:31.582796 2581 generic.go:356] "Generic (PLEG): container finished" podID="96c15dbf-c175-42ce-a0f1-3d74224b66ba" containerID="5ed44cb73485876dc07ca5f65f00efe8b6d26455193ec7a93bdf67aac977fd17" exitCode=0 Sep 14 05:21:31.583232 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:31.582872 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" event={"ID":"96c15dbf-c175-42ce-a0f1-3d74224b66ba","Type":"ContainerDied","Data":"5ed44cb73485876dc07ca5f65f00efe8b6d26455193ec7a93bdf67aac977fd17"} Sep 14 05:21:32.725731 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:32.725705 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:32.846818 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:32.846737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca\") pod \"96c15dbf-c175-42ce-a0f1-3d74224b66ba\" (UID: \"96c15dbf-c175-42ce-a0f1-3d74224b66ba\") " Sep 14 05:21:32.847085 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:32.847061 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96c15dbf-c175-42ce-a0f1-3d74224b66ba" (UID: "96c15dbf-c175-42ce-a0f1-3d74224b66ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:32.948383 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:32.948355 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c15dbf-c175-42ce-a0f1-3d74224b66ba-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:33.593734 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:33.593696 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" event={"ID":"96c15dbf-c175-42ce-a0f1-3d74224b66ba","Type":"ContainerDied","Data":"9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0"} Sep 14 05:21:33.593734 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:33.593727 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj" Sep 14 05:21:33.593907 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:33.593750 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9261e318c1a5ddaf04d73e8117aee4609f7b8c35984e4861e798572c5250c2f0" Sep 14 05:21:40.047882 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.047846 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck"] Sep 14 05:21:40.049800 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.049772 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2ff4a6-29822715-rj2ck"] Sep 14 05:21:40.077369 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.077333 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:21:40.077583 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.077560 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" podUID="a6d78c1f-56bc-4051-8783-45f143f48749" containerName="payload-processing" containerID="cri-o://9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc" gracePeriod=30 Sep 14 05:21:40.086715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.086686 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:21:40.086944 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.086922 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" podUID="80fd60b5-b337-40e4-83c7-d0beb1227996" containerName="payload-pre-processing" containerID="cri-o://075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8" gracePeriod=30 Sep 14 05:21:40.285244 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.284706 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj"] Sep 14 05:21:40.289288 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.289262 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2ff4a6-pvqxj"] Sep 14 05:21:40.355011 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.354985 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:21:40.376030 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.376008 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:21:40.407492 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.407468 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume\") pod \"80fd60b5-b337-40e4-83c7-d0beb1227996\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " Sep 14 05:21:40.407612 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.407574 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kgwq\" (UniqueName: \"kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq\") pod \"a6d78c1f-56bc-4051-8783-45f143f48749\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " Sep 14 05:21:40.407687 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.407632 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume\") pod \"a6d78c1f-56bc-4051-8783-45f143f48749\" (UID: \"a6d78c1f-56bc-4051-8783-45f143f48749\") " Sep 14 05:21:40.407687 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.407650 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6df2\" (UniqueName: \"kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2\") pod \"80fd60b5-b337-40e4-83c7-d0beb1227996\" (UID: \"80fd60b5-b337-40e4-83c7-d0beb1227996\") " Sep 14 05:21:40.407839 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.407811 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80fd60b5-b337-40e4-83c7-d0beb1227996" (UID: "80fd60b5-b337-40e4-83c7-d0beb1227996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:40.408076 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.408053 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6d78c1f-56bc-4051-8783-45f143f48749" (UID: "a6d78c1f-56bc-4051-8783-45f143f48749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:40.409699 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.409670 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq" (OuterVolumeSpecName: "kube-api-access-4kgwq") pod "a6d78c1f-56bc-4051-8783-45f143f48749" (UID: "a6d78c1f-56bc-4051-8783-45f143f48749"). InnerVolumeSpecName "kube-api-access-4kgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:40.409793 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.409728 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2" (OuterVolumeSpecName: "kube-api-access-r6df2") pod "80fd60b5-b337-40e4-83c7-d0beb1227996" (UID: "80fd60b5-b337-40e4-83c7-d0beb1227996"). InnerVolumeSpecName "kube-api-access-r6df2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:40.508624 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.508591 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kgwq\" (UniqueName: \"kubernetes.io/projected/a6d78c1f-56bc-4051-8783-45f143f48749-kube-api-access-4kgwq\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:40.508624 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.508618 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d78c1f-56bc-4051-8783-45f143f48749-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:40.508624 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.508630 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6df2\" (UniqueName: \"kubernetes.io/projected/80fd60b5-b337-40e4-83c7-d0beb1227996-kube-api-access-r6df2\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:40.508853 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.508642 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fd60b5-b337-40e4-83c7-d0beb1227996-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:40.628712 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.628679 2581 generic.go:356] "Generic (PLEG): container finished" podID="a6d78c1f-56bc-4051-8783-45f143f48749" containerID="9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc" exitCode=0 Sep 14 05:21:40.628878 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.628729 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" Sep 14 05:21:40.628878 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.628765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" event={"ID":"a6d78c1f-56bc-4051-8783-45f143f48749","Type":"ContainerDied","Data":"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc"} Sep 14 05:21:40.628878 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.628825 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd" event={"ID":"a6d78c1f-56bc-4051-8783-45f143f48749","Type":"ContainerDied","Data":"2429f7fe10a7bd8a176a49de48c1519b6858112b16f8b5b51855edeb807985c0"} Sep 14 05:21:40.628878 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.628852 2581 scope.go:117] "RemoveContainer" containerID="9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc" Sep 14 05:21:40.630148 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.630103 2581 generic.go:356] "Generic (PLEG): container finished" podID="80fd60b5-b337-40e4-83c7-d0beb1227996" containerID="075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8" exitCode=0 Sep 14 05:21:40.630267 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.630152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" event={"ID":"80fd60b5-b337-40e4-83c7-d0beb1227996","Type":"ContainerDied","Data":"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8"} Sep 14 05:21:40.630267 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.630204 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" event={"ID":"80fd60b5-b337-40e4-83c7-d0beb1227996","Type":"ContainerDied","Data":"b4ddd176aa7e4771505459bc7f3b1367e45448d3eb9c7690ceb4324c0c5e3eff"} Sep 14 05:21:40.630267 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.630166 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr" Sep 14 05:21:40.645715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.645697 2581 scope.go:117] "RemoveContainer" containerID="9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc" Sep 14 05:21:40.645976 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:40.645959 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc\": container with ID starting with 9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc not found: ID does not exist" containerID="9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc" Sep 14 05:21:40.646023 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.645982 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc"} err="failed to get container status \"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc\": rpc error: code = NotFound desc = could not find container \"9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc\": container with ID starting with 9cae444cb57ba9c97485e77a93828cfcf6b58fcdf91eb6b1fbce317c7edb65cc not found: ID does not exist" Sep 14 05:21:40.646023 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.645998 2581 scope.go:117] "RemoveContainer" containerID="075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8" Sep 14 05:21:40.653876 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.653859 2581 scope.go:117] "RemoveContainer" containerID="075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8" Sep 14 05:21:40.654111 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:40.654096 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8\": container with ID starting with 075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8 not found: ID does not exist" containerID="075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8" Sep 14 05:21:40.654175 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.654129 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8"} err="failed to get container status \"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8\": rpc error: code = NotFound desc = could not find container \"075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8\": container with ID starting with 075f30b12cae8b5d0f5c7933139db77662b9a7f5cc599833769cb1de978235c8 not found: ID does not exist" Sep 14 05:21:40.657224 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.657201 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:21:40.663848 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.663827 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2ff4a6-78bb8d9894-28fgr"] Sep 14 05:21:40.674490 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.674469 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:21:40.678377 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:40.678355 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2ff4a6-7859858b49-5nqtd"] Sep 14 05:21:41.552811 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:41.552784 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80fd60b5-b337-40e4-83c7-d0beb1227996" path="/var/lib/kubelet/pods/80fd60b5-b337-40e4-83c7-d0beb1227996/volumes" Sep 14 05:21:41.553187 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:41.553135 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c15dbf-c175-42ce-a0f1-3d74224b66ba" path="/var/lib/kubelet/pods/96c15dbf-c175-42ce-a0f1-3d74224b66ba/volumes" Sep 14 05:21:41.553467 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:41.553453 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d78c1f-56bc-4051-8783-45f143f48749" path="/var/lib/kubelet/pods/a6d78c1f-56bc-4051-8783-45f143f48749/volumes" Sep 14 05:21:41.553696 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:41.553686 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31e50f0-7922-481a-b8e6-324411da33a2" path="/var/lib/kubelet/pods/b31e50f0-7922-481a-b8e6-324411da33a2/volumes" Sep 14 05:21:47.604442 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604408 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th"] Sep 14 05:21:47.604799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604789 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d78c1f-56bc-4051-8783-45f143f48749" containerName="payload-processing" Sep 14 05:21:47.604799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604800 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d78c1f-56bc-4051-8783-45f143f48749" containerName="payload-processing" Sep 14 05:21:47.604879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604813 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fd60b5-b337-40e4-83c7-d0beb1227996" containerName="payload-pre-processing" Sep 14 05:21:47.604879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604819 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fd60b5-b337-40e4-83c7-d0beb1227996" containerName="payload-pre-processing" Sep 14 05:21:47.604879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604834 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c15dbf-c175-42ce-a0f1-3d74224b66ba" containerName="revoke-keys" Sep 14 05:21:47.604879 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604840 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c15dbf-c175-42ce-a0f1-3d74224b66ba" containerName="revoke-keys" Sep 14 05:21:47.605013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604892 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fd60b5-b337-40e4-83c7-d0beb1227996" containerName="payload-pre-processing" Sep 14 05:21:47.605013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604902 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c15dbf-c175-42ce-a0f1-3d74224b66ba" containerName="revoke-keys" Sep 14 05:21:47.605013 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.604908 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d78c1f-56bc-4051-8783-45f143f48749" containerName="payload-processing" Sep 14 05:21:47.609270 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.609254 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:47.611244 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.611223 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:21:47.611337 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.611225 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\"" Sep 14 05:21:47.615217 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.615195 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th"] Sep 14 05:21:47.673513 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.673473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th\" (UID: \"2b15d2db-3768-4a37-b6b4-376dc72acf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:47.773952 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.773925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th\" (UID: \"2b15d2db-3768-4a37-b6b4-376dc72acf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:47.774488 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.774471 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th\" (UID: \"2b15d2db-3768-4a37-b6b4-376dc72acf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:47.921239 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:47.921159 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:48.194838 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:48.194762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th"] Sep 14 05:21:48.198380 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:21:48.198351 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b15d2db_3768_4a37_b6b4_376dc72acf2c.slice/crio-33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057 WatchSource:0}: Error finding container 33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057: Status 404 returned error can't find the container with id 33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057 Sep 14 05:21:48.674629 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:48.674597 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" event={"ID":"2b15d2db-3768-4a37-b6b4-376dc72acf2c","Type":"ContainerStarted","Data":"7efc4918d40eeb7a47c6fb350c9325effa33b62a97fe2009552d80e8794b1da0"} Sep 14 05:21:48.674961 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:48.674652 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" event={"ID":"2b15d2db-3768-4a37-b6b4-376dc72acf2c","Type":"ContainerStarted","Data":"33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057"} Sep 14 05:21:48.689843 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:48.689797 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" podStartSLOduration=1.689783378 podStartE2EDuration="1.689783378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:21:48.688327388 +0000 UTC m=+2119.645944127" watchObservedRunningTime="2026-09-14 05:21:48.689783378 +0000 UTC m=+2119.647400115" Sep 14 05:21:49.682715 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:49.682676 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b15d2db-3768-4a37-b6b4-376dc72acf2c" containerID="7efc4918d40eeb7a47c6fb350c9325effa33b62a97fe2009552d80e8794b1da0" exitCode=0 Sep 14 05:21:49.683109 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:49.682758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" event={"ID":"2b15d2db-3768-4a37-b6b4-376dc72acf2c","Type":"ContainerDied","Data":"7efc4918d40eeb7a47c6fb350c9325effa33b62a97fe2009552d80e8794b1da0"} Sep 14 05:21:50.819209 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:50.819182 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:50.908086 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:50.908061 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca\") pod \"2b15d2db-3768-4a37-b6b4-376dc72acf2c\" (UID: \"2b15d2db-3768-4a37-b6b4-376dc72acf2c\") " Sep 14 05:21:50.908403 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:50.908379 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b15d2db-3768-4a37-b6b4-376dc72acf2c" (UID: "2b15d2db-3768-4a37-b6b4-376dc72acf2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:51.009146 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:51.009052 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b15d2db-3768-4a37-b6b4-376dc72acf2c-maas-api-service-ca\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:51.694583 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:51.694554 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" Sep 14 05:21:51.694732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:51.694558 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th" event={"ID":"2b15d2db-3768-4a37-b6b4-376dc72acf2c","Type":"ContainerDied","Data":"33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057"} Sep 14 05:21:51.694732 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:51.694680 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33dea2adaf5dacdd5c2f282958da2ccf59deab9ca822d5dfad6fcfaac70be057" Sep 14 05:21:58.134512 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.134475 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:21:58.134967 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.134725 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" podUID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" containerName="payload-processing" containerID="cri-o://3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c" gracePeriod=30 Sep 14 05:21:58.140040 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.140014 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:21:58.140438 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.140408 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" podUID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" containerName="payload-pre-processing" containerID="cri-o://616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab" gracePeriod=30 Sep 14 05:21:58.337407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.337344 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th"] Sep 14 05:21:58.340021 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.339997 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-27be50-w49th"] Sep 14 05:21:58.434579 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.434550 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:21:58.442242 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.442218 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:21:58.577370 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577338 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume\") pod \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " Sep 14 05:21:58.577545 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577433 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzt7p\" (UniqueName: \"kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p\") pod \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " Sep 14 05:21:58.577589 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577537 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn4r4\" (UniqueName: \"kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4\") pod \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\" (UID: \"8cb6493b-567a-4964-b3c5-ffb2bc92fd66\") " Sep 14 05:21:58.577644 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577585 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume\") pod \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\" (UID: \"49527fdb-f6f0-4da6-b6c0-edb2f9d56099\") " Sep 14 05:21:58.577770 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577750 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cb6493b-567a-4964-b3c5-ffb2bc92fd66" (UID: "8cb6493b-567a-4964-b3c5-ffb2bc92fd66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:58.577973 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.577949 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49527fdb-f6f0-4da6-b6c0-edb2f9d56099" (UID: "49527fdb-f6f0-4da6-b6c0-edb2f9d56099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 05:21:58.578099 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.578081 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:58.578190 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.578101 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-plugins-config-volume\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:58.579582 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.579558 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p" (OuterVolumeSpecName: "kube-api-access-nzt7p") pod "49527fdb-f6f0-4da6-b6c0-edb2f9d56099" (UID: "49527fdb-f6f0-4da6-b6c0-edb2f9d56099"). InnerVolumeSpecName "kube-api-access-nzt7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:58.579659 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.579603 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4" (OuterVolumeSpecName: "kube-api-access-pn4r4") pod "8cb6493b-567a-4964-b3c5-ffb2bc92fd66" (UID: "8cb6493b-567a-4964-b3c5-ffb2bc92fd66"). InnerVolumeSpecName "kube-api-access-pn4r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 05:21:58.679509 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.679442 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzt7p\" (UniqueName: \"kubernetes.io/projected/49527fdb-f6f0-4da6-b6c0-edb2f9d56099-kube-api-access-nzt7p\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:58.679509 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.679469 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4r4\" (UniqueName: \"kubernetes.io/projected/8cb6493b-567a-4964-b3c5-ffb2bc92fd66-kube-api-access-pn4r4\") on node \"ip-10-0-132-238.ec2.internal\" DevicePath \"\"" Sep 14 05:21:58.733768 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.733737 2581 generic.go:356] "Generic (PLEG): container finished" podID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" containerID="3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c" exitCode=0 Sep 14 05:21:58.733927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.733785 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" Sep 14 05:21:58.733927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.733828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" event={"ID":"49527fdb-f6f0-4da6-b6c0-edb2f9d56099","Type":"ContainerDied","Data":"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c"} Sep 14 05:21:58.733927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.733888 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl" event={"ID":"49527fdb-f6f0-4da6-b6c0-edb2f9d56099","Type":"ContainerDied","Data":"3d3bab440e1666601ce910ec8788eafbdb1d3b7c4c66bab90920b18e139a37e2"} Sep 14 05:21:58.733927 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.733915 2581 scope.go:117] "RemoveContainer" containerID="3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c" Sep 14 05:21:58.735350 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.735330 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" containerID="616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab" exitCode=0 Sep 14 05:21:58.735445 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.735376 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" Sep 14 05:21:58.735504 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.735375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" event={"ID":"8cb6493b-567a-4964-b3c5-ffb2bc92fd66","Type":"ContainerDied","Data":"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab"} Sep 14 05:21:58.735567 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.735521 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn" event={"ID":"8cb6493b-567a-4964-b3c5-ffb2bc92fd66","Type":"ContainerDied","Data":"55926a55aa70aa1770c1c0a55762d1771e156fea0604c7c790738b9d3a8eee4e"} Sep 14 05:21:58.743997 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.743979 2581 scope.go:117] "RemoveContainer" containerID="3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c" Sep 14 05:21:58.744320 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:58.744301 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c\": container with ID starting with 3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c not found: ID does not exist" containerID="3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c" Sep 14 05:21:58.744406 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.744327 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c"} err="failed to get container status \"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c\": rpc error: code = NotFound desc = could not find container \"3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c\": container with ID starting with 3ae110fcc1e2326d5315285b373dc00cb3170b4a74c396a90d8abf79a651258c not found: ID does not exist" Sep 14 05:21:58.744406 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.744343 2581 scope.go:117] "RemoveContainer" containerID="616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab" Sep 14 05:21:58.753057 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.753040 2581 scope.go:117] "RemoveContainer" containerID="616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab" Sep 14 05:21:58.753325 ip-10-0-132-238 kubenswrapper[2581]: E0914 05:21:58.753307 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab\": container with ID starting with 616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab not found: ID does not exist" containerID="616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab" Sep 14 05:21:58.753371 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.753334 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab"} err="failed to get container status \"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab\": rpc error: code = NotFound desc = could not find container \"616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab\": container with ID starting with 616ce0ebf72989a45fad427092609f5bd4173d39edc3b5a9a10308a3029b42ab not found: ID does not exist" Sep 14 05:21:58.757578 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.757556 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:21:58.759602 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.759579 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-27be50-6565465c95-lmdjl"] Sep 14 05:21:58.768338 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.768317 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:21:58.775483 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:58.775457 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-27be50-599698f7b6-87fpn"] Sep 14 05:21:59.552131 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:59.552098 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b15d2db-3768-4a37-b6b4-376dc72acf2c" path="/var/lib/kubelet/pods/2b15d2db-3768-4a37-b6b4-376dc72acf2c/volumes" Sep 14 05:21:59.552494 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:59.552480 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" path="/var/lib/kubelet/pods/49527fdb-f6f0-4da6-b6c0-edb2f9d56099/volumes" Sep 14 05:21:59.552809 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:21:59.552797 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" path="/var/lib/kubelet/pods/8cb6493b-567a-4964-b3c5-ffb2bc92fd66/volumes" Sep 14 05:22:29.983848 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:22:29.983769 2581 scope.go:117] "RemoveContainer" containerID="1f2220db74d631fabb04f874127c4c440912c2336fd2293f8ac6ae99f223c9fc" Sep 14 05:26:30.039870 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:26:30.039826 2581 scope.go:117] "RemoveContainer" containerID="54b1ca8989bb63f3c93577304139a65cabddc9b2d879433de844fbda7ae6a89d" Sep 14 05:27:50.441928 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.441883 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n"] Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442309 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" containerName="payload-pre-processing" Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442325 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" containerName="payload-pre-processing" Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442338 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15d2db-3768-4a37-b6b4-376dc72acf2c" containerName="revoke-keys" Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442344 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15d2db-3768-4a37-b6b4-376dc72acf2c" containerName="revoke-keys" Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442368 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" containerName="payload-processing" Sep 14 05:27:50.442430 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442375 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" containerName="payload-processing" Sep 14 05:27:50.442656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442444 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb6493b-567a-4964-b3c5-ffb2bc92fd66" containerName="payload-pre-processing" Sep 14 05:27:50.442656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442452 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b15d2db-3768-4a37-b6b4-376dc72acf2c" containerName="revoke-keys" Sep 14 05:27:50.442656 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.442458 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49527fdb-f6f0-4da6-b6c0-edb2f9d56099" containerName="payload-processing" Sep 14 05:27:50.445606 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.445579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.447684 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.447665 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 05:27:50.447684 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.447681 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 05:27:50.447869 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.447690 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9nv8\"" Sep 14 05:27:50.447869 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.447738 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 05:27:50.447869 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.447770 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 05:27:50.454256 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.454230 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n"] Sep 14 05:27:50.559677 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.559648 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjkl\" (UniqueName: \"kubernetes.io/projected/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-kube-api-access-7cjkl\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.559677 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.559680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-openshift-service-ca\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.559893 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.559751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-maas-api-tls\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.559893 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.559786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-metrics-certs\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.661091 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.661054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-maas-api-tls\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.661328 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.661157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-metrics-certs\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.661328 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.661226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjkl\" (UniqueName: \"kubernetes.io/projected/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-kube-api-access-7cjkl\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.661328 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.661255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-openshift-service-ca\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.662006 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.661979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-openshift-service-ca\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.663555 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.663536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-maas-api-tls\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.663686 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.663667 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-metrics-certs\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.668799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.668776 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjkl\" (UniqueName: \"kubernetes.io/projected/d1d16ab1-ea45-4126-9260-bbf47cb4ee7c-kube-api-access-7cjkl\") pod \"maas-api-66f5596c4d-5tp6n\" (UID: \"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c\") " pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:50.757358 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:50.757276 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:51.032479 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:51.032380 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n"] Sep 14 05:27:51.035755 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:27:51.035729 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d16ab1_ea45_4126_9260_bbf47cb4ee7c.slice/crio-f962fb2d923090a653c22f129fbda78a4d11a5ef1a904dd5547063e572dda9dd WatchSource:0}: Error finding container f962fb2d923090a653c22f129fbda78a4d11a5ef1a904dd5547063e572dda9dd: Status 404 returned error can't find the container with id f962fb2d923090a653c22f129fbda78a4d11a5ef1a904dd5547063e572dda9dd Sep 14 05:27:51.037603 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:51.037583 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 05:27:51.501704 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:51.501666 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" event={"ID":"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c","Type":"ContainerStarted","Data":"f962fb2d923090a653c22f129fbda78a4d11a5ef1a904dd5547063e572dda9dd"} Sep 14 05:27:53.512912 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:53.512876 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" event={"ID":"d1d16ab1-ea45-4126-9260-bbf47cb4ee7c","Type":"ContainerStarted","Data":"3ef1f924677b540ccbfc0d68195834a7f91e05edb9f5bcc7da199f8a7d03cfee"} Sep 14 05:27:53.513393 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:53.513021 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:27:53.531889 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:53.531836 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" podStartSLOduration=1.17396814 podStartE2EDuration="3.531819737s" totalImagesPullingTime="2.357851597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:27:53.530398994 +0000 UTC m=+2484.488015760" watchObservedRunningTime="2026-09-14 05:27:53.531819737 +0000 UTC m=+2484.489436473" Sep 14 05:27:59.523422 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:27:59.523395 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66f5596c4d-5tp6n" Sep 14 05:28:18.825051 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:18.825001 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-q4kkd_0bec66f9-3d2e-4dd7-bb7b-daa71e966e45/manager/0.log" Sep 14 05:28:19.191765 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:19.191743 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-xh497_fe6c4506-3bff-4596-a594-d29c984f92d5/copy-manifests/0.log" Sep 14 05:28:19.198977 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:19.198954 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-xh497_fe6c4506-3bff-4596-a594-d29c984f92d5/manager/0.log" Sep 14 05:28:19.552536 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:19.552455 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66f5596c4d-5tp6n_d1d16ab1-ea45-4126-9260-bbf47cb4ee7c/maas-api/0.log" Sep 14 05:28:21.992089 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:21.991997 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d_0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23/istio-proxy/0.log" Sep 14 05:28:22.107272 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:22.107234 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h564w_259a958e-0ad5-42c1-9395-051d7da3ab9d/discovery/0.log" Sep 14 05:28:22.447842 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:22.447811 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/1.log" Sep 14 05:28:22.681697 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:22.681670 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:28:23.248749 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:23.248715 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ftw7w_4e33bb41-be6f-451b-81a5-0d3d090c5fca/main/0.log" Sep 14 05:28:23.362953 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:23.362913 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-hrsjm_ccbdd262-0ce1-4c26-b4ec-83c3cc0c1296/main/0.log" Sep 14 05:28:23.703778 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:23.703748 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cwgkj_514d723b-7e28-4d28-97a4-54dc77771d02/main/0.log" Sep 14 05:28:30.067051 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:30.067020 2581 scope.go:117] "RemoveContainer" containerID="7efc4918d40eeb7a47c6fb350c9325effa33b62a97fe2009552d80e8794b1da0" Sep 14 05:28:30.074610 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:30.074593 2581 scope.go:117] "RemoveContainer" containerID="5ed44cb73485876dc07ca5f65f00efe8b6d26455193ec7a93bdf67aac977fd17" Sep 14 05:28:32.391267 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:32.391235 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lw9b4_2e465651-931b-4cc0-9fc3-f3eee8ce3ebd/global-pull-secret-syncer/0.log" Sep 14 05:28:32.564708 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:32.564677 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m7964_b332df2d-27ca-4bb8-baad-7aadb1148973/konnectivity-agent/0.log" Sep 14 05:28:32.628905 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:32.628876 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-238.ec2.internal_729843ab636a52a35ed9dc96e1e6fe1d/haproxy/0.log" Sep 14 05:28:38.831178 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.831088 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/alertmanager/0.log" Sep 14 05:28:38.876819 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.876796 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/config-reloader/0.log" Sep 14 05:28:38.901498 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.901475 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/kube-rbac-proxy-web/0.log" Sep 14 05:28:38.924190 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.924170 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/kube-rbac-proxy/0.log" Sep 14 05:28:38.954388 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.954352 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/kube-rbac-proxy-metric/0.log" Sep 14 05:28:38.980874 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:38.980853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/prom-label-proxy/0.log" Sep 14 05:28:39.003646 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.003627 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21e57896-5485-45e4-b4c9-1f0248ef662d/init-config-reloader/0.log" Sep 14 05:28:39.089674 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.089622 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-6wdcz_2c180f68-2b72-4976-9254-4b55371887e3/kube-state-metrics/0.log" Sep 14 05:28:39.113738 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.113721 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-6wdcz_2c180f68-2b72-4976-9254-4b55371887e3/kube-rbac-proxy-main/0.log" Sep 14 05:28:39.137425 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.137409 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-6wdcz_2c180f68-2b72-4976-9254-4b55371887e3/kube-rbac-proxy-self/0.log" Sep 14 05:28:39.173742 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.173723 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77d5df6575-79nvs_b46b88f2-c4f2-4169-ac2a-1a91d773d637/metrics-server/0.log" Sep 14 05:28:39.240588 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.240570 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fmds_3f4f03ef-d501-4fa7-8643-9e04cea45767/node-exporter/0.log" Sep 14 05:28:39.265685 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.265664 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fmds_3f4f03ef-d501-4fa7-8643-9e04cea45767/kube-rbac-proxy/0.log" Sep 14 05:28:39.289711 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.289694 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fmds_3f4f03ef-d501-4fa7-8643-9e04cea45767/init-textfile/0.log" Sep 14 05:28:39.517224 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.517202 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-x2zjr_f9379c23-2544-41fd-83dc-63896ef21f78/kube-rbac-proxy-main/0.log" Sep 14 05:28:39.559765 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.559738 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-x2zjr_f9379c23-2544-41fd-83dc-63896ef21f78/kube-rbac-proxy-self/0.log" Sep 14 05:28:39.602528 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.602485 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-x2zjr_f9379c23-2544-41fd-83dc-63896ef21f78/openshift-state-metrics/0.log" Sep 14 05:28:39.646057 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.645970 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/prometheus/0.log" Sep 14 05:28:39.677753 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.677712 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/config-reloader/0.log" Sep 14 05:28:39.702369 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.702347 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/thanos-sidecar/0.log" Sep 14 05:28:39.726872 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.726851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/kube-rbac-proxy-web/0.log" Sep 14 05:28:39.757269 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.757250 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/kube-rbac-proxy/0.log" Sep 14 05:28:39.783270 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.783241 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/kube-rbac-proxy-thanos/0.log" Sep 14 05:28:39.809669 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.809639 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45d054fb-2a11-4d82-b0ad-87ee0ae0ba50/init-config-reloader/0.log" Sep 14 05:28:39.901652 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:39.901566 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-9j76s_db24ee8c-1786-4098-b19a-f6c7b16ed9d1/prometheus-operator-admission-webhook/0.log" Sep 14 05:28:42.493863 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.493819 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh"] Sep 14 05:28:42.497616 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.497598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.499741 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.499713 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvbqd\"/\"openshift-service-ca.crt\"" Sep 14 05:28:42.500039 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.500021 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvbqd\"/\"kube-root-ca.crt\"" Sep 14 05:28:42.500161 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.500022 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvbqd\"/\"default-dockercfg-s74lc\"" Sep 14 05:28:42.506285 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.506264 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh"] Sep 14 05:28:42.668568 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.668527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-podres\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.668799 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.668772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kznt\" (UniqueName: \"kubernetes.io/projected/413b8ca0-da53-4b3f-8162-147a91183df1-kube-api-access-2kznt\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.668891 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.668870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-proc\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.668958 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.668925 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-lib-modules\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.669041 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.668992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-sys\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.731231 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.731203 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-26rmh_b32376a7-aaf0-4d64-a110-73fdc633589d/download-server/0.log" Sep 14 05:28:42.770374 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-podres\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kznt\" (UniqueName: \"kubernetes.io/projected/413b8ca0-da53-4b3f-8162-147a91183df1-kube-api-access-2kznt\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-proc\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-podres\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-lib-modules\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770533 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-sys\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770750 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770561 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-sys\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770750 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770520 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-proc\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.770750 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.770583 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/413b8ca0-da53-4b3f-8162-147a91183df1-lib-modules\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.778499 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.778475 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kznt\" (UniqueName: \"kubernetes.io/projected/413b8ca0-da53-4b3f-8162-147a91183df1-kube-api-access-2kznt\") pod \"perf-node-gather-daemonset-hm2qh\" (UID: \"413b8ca0-da53-4b3f-8162-147a91183df1\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:42.808950 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:42.808925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:43.079548 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.079523 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh"] Sep 14 05:28:43.081560 ip-10-0-132-238 kubenswrapper[2581]: W0914 05:28:43.081528 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod413b8ca0_da53_4b3f_8162_147a91183df1.slice/crio-14fecd807633cdaed284c9f68a03fb5b58f5427fac10c0486970dd91a3e9bd16 WatchSource:0}: Error finding container 14fecd807633cdaed284c9f68a03fb5b58f5427fac10c0486970dd91a3e9bd16: Status 404 returned error can't find the container with id 14fecd807633cdaed284c9f68a03fb5b58f5427fac10c0486970dd91a3e9bd16 Sep 14 05:28:43.416988 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.416955 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-qfjww_98604993-7109-4566-bbf4-9e909393f78e/volume-data-source-validator/0.log" Sep 14 05:28:43.768017 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.767934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" event={"ID":"413b8ca0-da53-4b3f-8162-147a91183df1","Type":"ContainerStarted","Data":"47761757bdabba6d6c8a261e1ff3360c554bfe376b15fd16364228c75264b60c"} Sep 14 05:28:43.768017 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.767990 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:43.768017 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.768012 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" event={"ID":"413b8ca0-da53-4b3f-8162-147a91183df1","Type":"ContainerStarted","Data":"14fecd807633cdaed284c9f68a03fb5b58f5427fac10c0486970dd91a3e9bd16"} Sep 14 05:28:43.782928 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:43.782886 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" podStartSLOduration=1.7828722080000001 podStartE2EDuration="1.782872208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 05:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:28:43.782847258 +0000 UTC m=+2534.740464006" watchObservedRunningTime="2026-09-14 05:28:43.782872208 +0000 UTC m=+2534.740488943" Sep 14 05:28:44.610280 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:44.610235 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-479w5_37cdc47c-6d0b-43e0-937e-a413fdb46d11/dns/0.log" Sep 14 05:28:44.632300 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:44.632273 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-479w5_37cdc47c-6d0b-43e0-937e-a413fdb46d11/kube-rbac-proxy/0.log" Sep 14 05:28:44.735133 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:44.735091 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwbjw_b10002bc-68a5-479a-a081-6e7f8a6cccb4/dns-node-resolver/0.log" Sep 14 05:28:45.342625 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:45.342600 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-445p5_f3371066-7823-4f23-95db-b5d47a1ce426/node-ca/0.log" Sep 14 05:28:46.310858 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:46.310823 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d_0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23/istio-proxy/0.log" Sep 14 05:28:46.691377 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:46.691340 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h564w_259a958e-0ad5-42c1-9395-051d7da3ab9d/discovery/0.log" Sep 14 05:28:46.773470 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:46.773432 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/0.log" Sep 14 05:28:46.776954 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:46.776928 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/1.log" Sep 14 05:28:46.835332 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:46.835303 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:28:47.792770 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:47.792744 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7jcz2_edd700df-95e3-489b-93a3-1e6ee162a412/serve-healthcheck-canary/0.log" Sep 14 05:28:48.336696 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:48.336666 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-pkk55_e3b45323-6c9b-4747-8665-d81e85a37840/insights-operator/0.log" Sep 14 05:28:48.338510 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:48.338492 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-pkk55_e3b45323-6c9b-4747-8665-d81e85a37840/insights-operator/1.log" Sep 14 05:28:48.509810 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:48.509776 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdq5h_c0273b0e-9540-4ef2-adee-fc134683dcd3/kube-rbac-proxy/0.log" Sep 14 05:28:48.531602 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:48.531580 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdq5h_c0273b0e-9540-4ef2-adee-fc134683dcd3/exporter/0.log" Sep 14 05:28:48.555541 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:48.555518 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdq5h_c0273b0e-9540-4ef2-adee-fc134683dcd3/extractor/0.log" Sep 14 05:28:49.783636 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:49.783602 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-hm2qh" Sep 14 05:28:51.003384 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:51.003349 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-q4kkd_0bec66f9-3d2e-4dd7-bb7b-daa71e966e45/manager/0.log" Sep 14 05:28:51.177565 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:51.177542 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-xh497_fe6c4506-3bff-4596-a594-d29c984f92d5/manager/0.log" Sep 14 05:28:51.198741 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:51.198723 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-xh497_fe6c4506-3bff-4596-a594-d29c984f92d5/copy-manifests/0.log" Sep 14 05:28:52.668609 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:52.668579 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5854c46596-jnksg_98ec6ce5-0cff-4b36-a85d-735c65f359df/manager/0.log" Sep 14 05:28:52.844930 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:52.844902 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d_0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23/istio-proxy/0.log" Sep 14 05:28:53.212922 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:53.212894 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h564w_259a958e-0ad5-42c1-9395-051d7da3ab9d/discovery/0.log" Sep 14 05:28:53.256830 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:53.256806 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/0.log" Sep 14 05:28:53.258896 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:53.258871 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/1.log" Sep 14 05:28:53.300500 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:53.300478 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:28:54.833171 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:54.833139 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h9d2d_0e6ae42e-2592-423c-8d2c-a7bfdfb9bb23/istio-proxy/0.log" Sep 14 05:28:55.221302 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:55.221254 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h564w_259a958e-0ad5-42c1-9395-051d7da3ab9d/discovery/0.log" Sep 14 05:28:55.279867 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:55.279840 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/0.log" Sep 14 05:28:55.282511 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:55.282491 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6jjp4_d72ccddb-21db-4e33-bd50-747d4be9d4de/istio-proxy/1.log" Sep 14 05:28:55.331339 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:55.331306 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5t7fb_057f9ac5-59ae-410b-8666-a7abd28e0120/payload-processing/0.log" Sep 14 05:28:59.461919 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.461887 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/kube-multus-additional-cni-plugins/0.log" Sep 14 05:28:59.483471 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.483442 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/egress-router-binary-copy/0.log" Sep 14 05:28:59.508959 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.508936 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/cni-plugins/0.log" Sep 14 05:28:59.531230 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.531201 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/bond-cni-plugin/0.log" Sep 14 05:28:59.554220 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.554198 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/routeoverride-cni/0.log" Sep 14 05:28:59.578805 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.578787 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/whereabouts-cni-bincopy/0.log" Sep 14 05:28:59.603167 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.603144 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8764_66f3737b-d741-4c5c-8788-f184f954777e/whereabouts-cni/0.log" Sep 14 05:28:59.825407 ip-10-0-132-238 kubenswrapper[2581]: I0914 05:28:59.825343 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jtpdv_a88d88d6-16f8-4645-9d28-1db5b39be0a9/kube-multus/0.log"