Sep 18 17:49:53.496470 ip-10-0-133-72 systemd[1]: Starting Kubernetes Kubelet... Sep 18 17:49:56.772672 ip-10-0-133-72 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:49:56.772672 ip-10-0-133-72 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 17:49:56.772672 ip-10-0-133-72 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:49:56.772672 ip-10-0-133-72 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 17:49:56.772672 ip-10-0-133-72 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:49:56.773477 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.772741 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 17:49:56.776679 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776665 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.776679 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776679 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776683 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776686 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776689 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776692 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776695 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776698 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776701 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776703 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776706 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776708 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776711 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776713 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776716 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776719 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776721 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776724 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776726 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776729 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.776743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776734 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776742 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776745 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776749 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776752 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776755 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776758 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776761 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776763 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776766 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776768 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776771 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776774 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776776 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776779 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776781 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776784 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776787 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776789 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.777256 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776792 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776794 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776796 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776799 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776801 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776804 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776806 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776809 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776811 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776814 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776816 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776819 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776821 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776824 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776826 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776830 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776832 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776835 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776838 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776841 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.777732 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776844 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776846 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776849 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776851 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776854 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776856 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776858 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776861 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776864 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776866 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776868 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776871 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776873 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776876 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776878 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776881 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776884 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776886 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776889 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776891 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.778231 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776894 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776896 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776899 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776903 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776906 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776909 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776912 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776917 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776920 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776923 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776927 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776930 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776934 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776937 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776940 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776943 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776947 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776951 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776954 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.778710 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.776957 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777634 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777640 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777644 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777647 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777650 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777652 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777655 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777657 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777660 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777663 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777666 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777669 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777672 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777674 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777677 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777679 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777682 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777685 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777687 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777690 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.779165 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777693 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777696 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777699 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777701 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777704 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777707 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777710 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777713 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777716 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777718 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777721 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777724 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777726 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777729 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777731 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777734 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777737 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777739 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777742 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.779699 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777745 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777747 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777750 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777753 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777755 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777757 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777760 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777763 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777765 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777768 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777770 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777773 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777777 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777780 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777782 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777787 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777791 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777794 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777798 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.780219 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777800 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777803 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777806 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777809 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777811 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777814 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777817 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777819 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777822 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777824 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777827 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777829 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777832 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777834 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777837 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777839 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777842 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777846 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777850 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.780692 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777853 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777855 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777858 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777860 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777863 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777865 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777869 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777871 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777874 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777877 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777879 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777883 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777886 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777889 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777892 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777895 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777897 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777900 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777902 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777905 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.781178 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777908 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.777910 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778286 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778294 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778297 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778299 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778304 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778308 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778311 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778314 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778318 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778321 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778324 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778327 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778330 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778332 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778335 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778338 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778341 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778344 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.781696 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778346 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778349 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778352 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778354 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778357 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778359 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778362 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778364 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778367 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778370 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778372 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778375 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778377 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778380 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778382 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778386 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778389 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778391 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778394 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778396 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.782202 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778399 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778401 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778404 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778407 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778410 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778413 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778415 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778417 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778421 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778423 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778426 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778429 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778432 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778434 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778437 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778439 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778442 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778445 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778448 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.782730 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778451 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778453 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778456 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778458 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778461 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778463 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778466 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778468 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778472 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778475 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778477 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778480 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778482 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778485 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778488 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778490 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778493 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778496 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778499 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778501 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.783240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778504 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778506 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778509 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778513 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778515 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778518 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778522 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778525 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778528 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778531 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778534 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778536 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778539 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778542 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778544 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778546 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778549 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778551 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778554 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778556 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.783725 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778560 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.778562 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778634 2570 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778668 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778679 2570 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778688 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778692 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778695 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778699 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778704 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778708 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778711 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778715 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778718 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778721 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778726 2570 flags.go:64] FLAG: --cgroup-root="" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778728 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778732 2570 flags.go:64] FLAG: --client-ca-file="" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778734 2570 flags.go:64] FLAG: --cloud-provider="external" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778737 2570 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778741 2570 flags.go:64] FLAG: --cluster-domain="" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778744 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778747 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778750 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 17:49:56.784248 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778753 2570 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778757 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778760 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778763 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778766 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778769 2570 flags.go:64] FLAG: --contention-profiling="false" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778772 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778775 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778780 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778783 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778787 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778789 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778792 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778795 2570 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778798 2570 flags.go:64] FLAG: --enable-server="true" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778801 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778805 2570 flags.go:64] FLAG: --event-burst="100" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778809 2570 flags.go:64] FLAG: --event-qps="50" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778812 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778818 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778821 2570 flags.go:64] FLAG: --eviction-hard="" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778825 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778828 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778833 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778836 2570 flags.go:64] FLAG: --eviction-soft="" Sep 18 17:49:56.784819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778839 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778842 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778844 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778847 2570 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778850 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778853 2570 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778856 2570 flags.go:64] FLAG: --feature-gates="" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778860 2570 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778863 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778865 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778869 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778871 2570 flags.go:64] FLAG: --healthz-port="10248" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778874 2570 flags.go:64] FLAG: --help="false" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778877 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-133-72.ec2.internal" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778880 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778885 2570 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778888 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778892 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778895 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778898 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778901 2570 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778904 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778906 2570 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 17:49:56.785805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778909 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778912 2570 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778915 2570 flags.go:64] FLAG: --kube-reserved="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778918 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778921 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778925 2570 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778928 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778932 2570 flags.go:64] FLAG: --lock-file="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778934 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778937 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778940 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778945 2570 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778962 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778966 2570 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778969 2570 flags.go:64] FLAG: --logging-format="text" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778971 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778975 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778978 2570 flags.go:64] FLAG: --manifest-url="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778981 2570 flags.go:64] FLAG: --manifest-url-header="" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778985 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778988 2570 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778991 2570 flags.go:64] FLAG: --max-pods="110" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778995 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.778998 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779002 2570 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 17:49:56.786848 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779005 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779008 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779011 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779014 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779021 2570 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779024 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779027 2570 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779030 2570 flags.go:64] FLAG: --pod-cidr="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779033 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779039 2570 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779041 2570 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779044 2570 flags.go:64] FLAG: --pods-per-core="0" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779048 2570 flags.go:64] FLAG: --port="10250" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779051 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779054 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc2d2e09aaf40556" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779058 2570 flags.go:64] FLAG: --qos-reserved="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779061 2570 flags.go:64] FLAG: --read-only-port="10255" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779064 2570 flags.go:64] FLAG: --register-node="true" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779067 2570 flags.go:64] FLAG: --register-with-taints="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779071 2570 flags.go:64] FLAG: --registry-burst="10" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779074 2570 flags.go:64] FLAG: --registry-qps="5" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779076 2570 flags.go:64] FLAG: --reserved-cpus="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779079 2570 flags.go:64] FLAG: --reserved-memory="" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779083 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779086 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 17:49:56.787925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779089 2570 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779092 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779095 2570 flags.go:64] FLAG: --runonce="false" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779098 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779101 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779104 2570 flags.go:64] FLAG: --seccomp-default="false" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779109 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779112 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779115 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779118 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779121 2570 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779124 2570 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779127 2570 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779129 2570 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779132 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779135 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779138 2570 flags.go:64] FLAG: --system-cgroups="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779141 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779146 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779149 2570 flags.go:64] FLAG: --tls-cert-file="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779152 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779156 2570 flags.go:64] FLAG: --tls-min-version="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779160 2570 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779162 2570 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779165 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 17:49:56.789009 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779168 2570 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779171 2570 flags.go:64] FLAG: --v="2" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779175 2570 flags.go:64] FLAG: --version="false" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779179 2570 flags.go:64] FLAG: --vmodule="" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779197 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779200 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779314 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779318 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779322 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779325 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779328 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779331 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779334 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779339 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779342 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779345 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779348 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779351 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779354 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779356 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.790021 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779361 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779364 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779367 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779370 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779373 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779376 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779379 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779381 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779384 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779388 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779390 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779393 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779395 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779398 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779401 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779403 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779406 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779408 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779411 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779414 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.790796 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779417 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779419 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779422 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779424 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779427 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779431 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779434 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779436 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779439 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779442 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779444 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779447 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779449 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779452 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779454 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779457 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779459 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779462 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779464 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779467 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.791401 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779469 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779473 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779476 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779479 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779481 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779484 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779486 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779489 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779491 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779494 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779498 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779502 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779505 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779508 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779511 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779513 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779516 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779520 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779523 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779525 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.792013 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779528 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779532 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779535 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779538 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779541 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779544 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779547 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779549 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779552 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779554 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779557 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779560 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779562 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779565 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779570 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779573 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779576 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779578 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779581 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779583 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.792551 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779586 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779588 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779591 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779593 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.779596 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.779601 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.787364 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.787383 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.787408 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.787417 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787508 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787515 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787520 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787524 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787529 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.793068 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787534 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787538 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787542 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787546 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787551 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787555 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787559 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787563 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787568 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787574 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787580 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787586 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787591 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787596 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787601 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787606 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787611 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787616 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787620 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787624 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.793626 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787629 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787633 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787637 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787642 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787647 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787651 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787656 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787660 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787665 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787669 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787673 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787677 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787681 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787686 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787691 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787695 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787699 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787703 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787709 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.794538 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787715 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787719 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787723 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787727 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787733 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787738 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787742 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787747 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787751 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787757 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787761 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787765 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787769 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787774 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787778 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787782 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787787 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787792 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787796 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787800 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.795207 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787805 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787809 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787814 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787818 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787822 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787827 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787831 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787835 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787840 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787844 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787848 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787852 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787856 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787861 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787865 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787869 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787874 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787878 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787882 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787887 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.795743 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787894 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787899 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787904 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787909 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787913 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787918 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787922 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787926 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787930 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787935 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787939 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787943 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787947 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787950 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.787955 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.796259 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.787963 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788140 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788147 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788154 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788160 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788165 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788170 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788175 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788179 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788201 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788205 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788210 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788214 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788218 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788222 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788227 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788231 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788236 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788240 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788245 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:49:56.796634 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788249 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788252 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788256 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788260 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788264 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788268 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788274 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788278 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788282 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788286 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788290 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788294 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788298 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788303 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788307 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788311 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788315 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788320 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788325 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:49:56.797133 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788329 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788333 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788337 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788341 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788346 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788350 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788354 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788358 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788362 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788367 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788372 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788376 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788380 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788384 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788389 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788393 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788397 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788402 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788406 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788412 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:49:56.797621 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788416 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788420 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788424 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788428 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788432 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788437 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788441 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788446 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788453 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788458 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788462 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788466 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788470 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788475 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788479 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788483 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788487 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788491 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788495 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788500 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:49:56.809768 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788504 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788508 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788512 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788517 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788521 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788526 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788530 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788534 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788538 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788542 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788546 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788553 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788559 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788564 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788568 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788572 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788577 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788581 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788585 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:49:56.810364 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788589 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:56.788594 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.788602 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.788749 2570 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.791469 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.792523 2570 server.go:1019] "Starting client certificate rotation" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.792626 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.792663 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.795889 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 17:49:56.810820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.797689 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 17:49:56.811877 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.811864 2570 log.go:25] "Validated CRI v1 runtime API" Sep 18 17:49:56.813269 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.813255 2570 log.go:25] "Validated CRI v1 image API" Sep 18 17:49:56.814064 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.814050 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 17:49:56.816220 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.816204 2570 fs.go:135] Filesystem UUIDs: map[4aae6adc-80ad-486c-a103-f4639fd4f179:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9bc94132-4134-4291-9191-301ef0fd6320:/dev/nvme0n1p4] Sep 18 17:49:56.816269 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.816220 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 17:49:56.820360 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.820259 2570 manager.go:216] Machine: {Timestamp:2026-09-18 17:49:56.819781943 +0000 UTC m=+1.912437041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097112 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9a91c0809183ad256d3e0422d59f SystemUUID:ec2f9a91-c080-9183-ad25-6d3e0422d59f BootID:71495221-2a13-4309-a2d7-23e7bcd99008 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:13:80:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:13:80:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:3b:dd:37:1f:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 17:49:56.820360 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.820355 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 17:49:56.820456 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.820420 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 17:49:56.821896 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.821876 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 17:49:56.822021 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.821899 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 17:49:56.822946 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.822937 2570 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 17:49:56.822986 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.822949 2570 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 17:49:56.822986 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.822957 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 17:49:56.822986 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.822967 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 17:49:56.824155 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.824145 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 17:49:56.827573 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827559 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 18 17:49:56.827836 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827826 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 17:49:56.827874 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827855 2570 kubelet.go:486] "Attempting to sync node with API server" Sep 18 17:49:56.827874 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827867 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 17:49:56.827932 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827885 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 17:49:56.827932 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827901 2570 kubelet.go:398] "Adding apiserver pod source" Sep 18 17:49:56.827932 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.827910 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 17:49:56.828214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.828200 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 17:49:56.828269 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.828226 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 17:49:56.829345 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.829322 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 17:49:56.831340 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.831317 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 17:49:56.832900 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.832879 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 17:49:56.832982 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.832911 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 17:49:56.834149 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834137 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834154 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834159 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834164 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834174 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834195 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834210 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834216 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834222 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 17:49:56.834227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834228 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 17:49:56.834454 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834237 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 17:49:56.834454 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834242 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 17:49:56.834454 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834265 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 17:49:56.834454 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.834270 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 17:49:56.837863 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.837849 2570 server.go:1268] "Started kubelet" Sep 18 17:49:56.837991 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.837950 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 17:49:56.838043 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.837974 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 17:49:56.838043 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.838012 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 17:49:56.838706 ip-10-0-133-72 systemd[1]: Started Kubernetes Kubelet. Sep 18 17:49:56.839244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.838901 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 17:49:56.839244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.838996 2570 server.go:310] "Adding debug handlers to kubelet server" Sep 18 17:49:56.843236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.843218 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 17:49:56.844081 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.844061 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-72.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 17:49:56.844167 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.844078 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 17:49:56.844167 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.844154 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 17:49:56.844945 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.844927 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 17:49:56.845005 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.844914 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 17:49:56.845056 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.845022 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:56.845469 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845453 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 17:49:56.845620 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845605 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 17:49:56.845707 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845623 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 17:49:56.845761 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845715 2570 reconstruct.go:97] "Volume reconstruction finished" Sep 18 17:49:56.845761 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845722 2570 reconciler.go:29] "Reconciler: start to sync state" Sep 18 17:49:56.845884 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845864 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 17:49:56.845884 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845884 2570 factory.go:55] Registering systemd factory Sep 18 17:49:56.845973 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.845893 2570 factory.go:223] Registration of the systemd container factory successfully Sep 18 17:49:56.846171 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.846158 2570 factory.go:153] Registering CRI-O factory Sep 18 17:49:56.846268 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.846173 2570 factory.go:223] Registration of the crio container factory successfully Sep 18 17:49:56.846268 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.846209 2570 factory.go:103] Registering Raw factory Sep 18 17:49:56.846268 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.846221 2570 manager.go:1196] Started watching for new ooms in manager Sep 18 17:49:56.846520 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.846510 2570 manager.go:349] Starting recovery of all containers Sep 18 17:49:56.849924 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.849900 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 17:49:56.850010 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.849972 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 17:49:56.871732 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.871617 2570 manager.go:354] Recovery completed Sep 18 17:49:56.873607 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.872475 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b987fecd7c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.837824448 +0000 UTC m=+1.930479525,LastTimestamp:2026-09-18 17:49:56.837824448 +0000 UTC m=+1.930479525,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.875661 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.875648 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:56.877682 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.877663 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:56.877752 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.877698 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:56.877752 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.877711 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:56.878266 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878245 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 17:49:56.878266 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878257 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:56.878266 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878263 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 17:49:56.878417 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878284 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 18 17:49:56.878970 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878951 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:56.879027 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.878986 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:56.879027 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.879001 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:56.880563 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.880550 2570 policy_none.go:49] "None policy: Start" Sep 18 17:49:56.880626 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.880566 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 17:49:56.880626 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.880576 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 17:49:56.882695 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.882684 2570 policy_none.go:47] "Start" Sep 18 17:49:56.888489 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.888416 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d03c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877681604 +0000 UTC m=+1.970336683,LastTimestamp:2026-09-18 17:49:56.877681604 +0000 UTC m=+1.970336683,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.900563 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.900506 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d5aa3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877703843 +0000 UTC m=+1.970358923,LastTimestamp:2026-09-18 17:49:56.877703843 +0000 UTC m=+1.970358923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.911087 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d8b7b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877716347 +0000 UTC m=+1.970371426,LastTimestamp:2026-09-18 17:49:56.877716347 +0000 UTC m=+1.970371426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.920907 2570 manager.go:337] "Starting Device Plugin manager" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.920949 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.920958 2570 server.go:87] "Starting device plugin registration server" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.921145 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.921166 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.921258 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.921329 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.921338 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.921807 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.921844 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:56.923236 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.922451 2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-72.ec2.internal.18d67b98824d03c4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d03c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877681604 +0000 UTC m=+1.970336683,LastTimestamp:2026-09-18 17:49:56.878969761 +0000 UTC m=+1.971624846,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.931270 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.931208 2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-72.ec2.internal.18d67b98824d5aa3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d5aa3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877703843 +0000 UTC m=+1.970358923,LastTimestamp:2026-09-18 17:49:56.878992845 +0000 UTC m=+1.971647925,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.931375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.931275 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8kqrb" Sep 18 17:49:56.934320 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.934293 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 17:49:56.935435 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.935417 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 17:49:56.935523 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.935458 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 17:49:56.935523 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.935478 2570 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 17:49:56.935625 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.935514 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 17:49:56.943253 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.943147 2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-72.ec2.internal.18d67b98824d8b7b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b98824d8b7b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-72.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.877716347 +0000 UTC m=+1.970371426,LastTimestamp:2026-09-18 17:49:56.879006554 +0000 UTC m=+1.971661636,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:56.943341 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.943264 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 17:49:56.946705 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:56.946690 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8kqrb" Sep 18 17:49:56.952899 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:56.952848 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-72.ec2.internal.18d67b9884ffc0b3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-72.ec2.internal,UID:ip-10-0-133-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-72.ec2.internal,},FirstTimestamp:2026-09-18 17:49:56.922949811 +0000 UTC m=+2.015604881,LastTimestamp:2026-09-18 17:49:56.922949811 +0000 UTC m=+2.015604881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-72.ec2.internal,}" Sep 18 17:49:57.021975 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.021957 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:57.022839 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.022793 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:57.022839 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.022823 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:57.022839 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.022838 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:57.023050 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.022864 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.035894 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.035874 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal"] Sep 18 17:49:57.035964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.035927 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:57.038737 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.038723 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:57.038822 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.038752 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:57.038822 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.038765 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:57.041090 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041075 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:57.041225 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041200 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.041280 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041231 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:57.041917 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041903 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:57.041981 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041924 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:57.041981 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041946 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:57.041981 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041955 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:57.042086 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.041927 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:57.042086 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.042020 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:57.044285 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.044271 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.044347 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.044295 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:49:57.045017 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.045002 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:49:57.045082 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.045027 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:49:57.045082 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.045041 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:49:57.068791 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.068773 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-72.ec2.internal\" not found" node="ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.069401 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.069377 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.069479 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.069406 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-72.ec2.internal\": node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.073048 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.073034 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-72.ec2.internal\" not found" node="ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.111552 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.111532 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.147172 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.147152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.147261 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.147177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.147261 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.147208 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edf4d681d04ec5f43785f7bccdc8c978-config\") pod \"kube-apiserver-proxy-ip-10-0-133-72.ec2.internal\" (UID: \"edf4d681d04ec5f43785f7bccdc8c978\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.211597 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.211574 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.248157 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.248261 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.248261 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.248261 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edf4d681d04ec5f43785f7bccdc8c978-config\") pod \"kube-apiserver-proxy-ip-10-0-133-72.ec2.internal\" (UID: \"edf4d681d04ec5f43785f7bccdc8c978\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.248385 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edf4d681d04ec5f43785f7bccdc8c978-config\") pod \"kube-apiserver-proxy-ip-10-0-133-72.ec2.internal\" (UID: \"edf4d681d04ec5f43785f7bccdc8c978\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.248385 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.248309 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46296604ae17500791f24b361b85e841-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal\" (UID: \"46296604ae17500791f24b361b85e841\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.311883 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.311828 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.372401 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.372383 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.377053 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.377033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.412722 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.412698 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.513172 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.513137 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.613705 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.613632 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.714231 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.714208 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.792728 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.792705 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 17:49:57.814785 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.814763 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-72.ec2.internal\" not found" Sep 18 17:49:57.817573 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.817556 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:49:57.828359 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.828343 2570 apiserver.go:52] "Watching apiserver" Sep 18 17:49:57.835916 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.835896 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 17:49:57.836224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.836204 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rpxdg","openshift-network-diagnostics/network-check-target-9vfvg","openshift-ovn-kubernetes/ovnkube-node-54nft","kube-system/global-pull-secret-syncer-58kvg","openshift-cluster-node-tuning-operator/tuned-d6n2d","openshift-dns/node-resolver-j6k5v","openshift-image-registry/node-ca-g6r9r","openshift-multus/multus-additional-cni-plugins-d2wmf","openshift-multus/network-metrics-daemon-vrn47","openshift-network-operator/iptables-alerter-whmld","kube-system/konnectivity-agent-p5t9l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk"] Sep 18 17:49:57.838494 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.838472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.840561 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.840529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:57.840653 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.840616 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:49:57.840983 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.840968 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 17:49:57.841098 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.841084 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.841160 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.841103 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 17:49:57.841160 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.841150 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w6jlq\"" Sep 18 17:49:57.841280 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.841240 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.842611 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.842591 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.844595 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844576 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 17:49:57.844688 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844615 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 17:49:57.844748 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844696 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.844748 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844706 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.844867 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844778 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xl4fg\"" Sep 18 17:49:57.844867 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844863 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 17:49:57.844941 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.844926 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 17:49:57.845074 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.845062 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 17:49:57.845116 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.845074 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.845416 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.845403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.845463 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.845452 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:49:57.847735 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.847720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.849835 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.849818 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.850323 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.850301 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.850424 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.850387 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p4ppk\"" Sep 18 17:49:57.850614 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.850600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:57.851810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-socket-dir-parent\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.851810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-script-lib\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.851810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-conf\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.851810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-host\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.851810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrm8c\" (UniqueName: \"kubernetes.io/projected/658fc186-df9a-4f0b-b582-0b727164d3cf-kube-api-access-zrm8c\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.852014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-bin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-multus\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851913 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-netd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-slash\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.851980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysconfig\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-dbus\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-log-socket\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-var-lib-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-node-log\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-bin\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-modprobe-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-systemd\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-system-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852318 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-multus-certs\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852340 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-etc-kubernetes\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-kube-api-access-7hr28\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852401 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-systemd-units\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovn-node-metrics-cert\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-hostroot\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:57.852600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-config\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-kubelet-config\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-tuned\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-tmp\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-k8s-cni-cncf-io\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-etc-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852800 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852811 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9hbx7\"" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-run\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-lib-modules\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-netns\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-systemd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-var-lib-kubelet\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852956 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-daemon-config\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.852978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cni-binary-copy\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853075 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-netns\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-conf-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-kubelet\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853100 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xnr\" (UniqueName: \"kubernetes.io/projected/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-kube-api-access-s5xnr\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853142 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-kubernetes\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cnibin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-kubelet\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-env-overrides\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-ovn\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.853827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.853289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-sys\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.855580 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.855562 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 17:49:57.855680 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.855627 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" Sep 18 17:49:57.855740 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.855693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.856203 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.856170 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.856328 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.856171 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 17:49:57.856328 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.856258 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.856328 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.856270 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jrzkw\"" Sep 18 17:49:57.857873 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.857854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bjq6x\"" Sep 18 17:49:57.857942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.857927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 17:49:57.858254 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.858239 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:57.858307 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.858295 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:49:57.858594 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.858581 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 17:49:57.860229 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.860213 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 17:49:57.860827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.860802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:57.863425 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.863405 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 17:49:57.863629 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.863614 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:57.864099 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.864041 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.864099 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.864091 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.864958 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.864934 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v5xp5\"" Sep 18 17:49:57.867109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.865845 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 17:49:57.867109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.865874 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kn2f8\"" Sep 18 17:49:57.867109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.866303 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal"] Sep 18 17:49:57.867109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.866416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.867109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.867074 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 17:49:57.868785 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.868767 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 17:49:57.868785 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.868782 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 17:49:57.868902 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.868781 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 17:49:57.868902 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.868805 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p8hm9\"" Sep 18 17:49:57.869650 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.869633 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 17:49:57.869857 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.869841 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal"] Sep 18 17:49:57.891588 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.891555 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ngnlk" Sep 18 17:49:57.901840 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.901825 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ngnlk" Sep 18 17:49:57.946442 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.946413 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 17:49:57.951319 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.951281 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 17:44:56 +0000 UTC" deadline="2028-05-17 01:13:22.843260966 +0000 UTC" Sep 18 17:49:57.951319 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.951318 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14551h23m24.891945889s" Sep 18 17:49:57.953450 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-systemd\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.953547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.953547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489888fa-2c2a-4342-a9bc-2dbe32192439-host-slash\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:57.953547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-system-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-multus-certs\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-etc-kubernetes\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-systemd\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-kube-api-access-7hr28\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-system-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-systemd-units\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-etc-kubernetes\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-multus-certs\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovn-node-metrics-cert\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-system-cni-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717ab396-8f0c-45aa-bd46-6d524b982255-hosts-file\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953712 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-systemd-units\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.953747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-hostroot\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-hostroot\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-config\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-kubelet-config\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-tuned\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-tmp\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98wj\" (UniqueName: \"kubernetes.io/projected/d154cf9d-64bf-413d-a602-5a590e06b22f-kube-api-access-h98wj\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953907 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-kubelet-config\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-k8s-cni-cncf-io\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-etc-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.953998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-run\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-lib-modules\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954006 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e1842d-4e29-463e-a581-3e102a65c66b-agent-certs\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-netns\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-systemd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.954413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-var-lib-kubelet\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-daemon-config\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954233 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-config\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cni-binary-copy\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-netns\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-run\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-conf-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954335 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-etc-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-k8s-cni-cncf-io\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954402 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954411 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-run-netns\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954430 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-lib-modules\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-conf-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954456 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-systemd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-kubelet\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954492 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-kubelet\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.955308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954530 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-run-netns\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xnr\" (UniqueName: \"kubernetes.io/projected/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-kube-api-access-s5xnr\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-kubernetes\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-var-lib-kubelet\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954586 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cnibin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-kubelet\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-env-overrides\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-host\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954707 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-ovn\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954767 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-daemon-config\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-sys\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798dj\" (UniqueName: \"kubernetes.io/projected/5974dc88-f16c-4843-b2a2-9309f55d5ecb-kube-api-access-798dj\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-socket-dir-parent\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.956224 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.954864 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-script-lib\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954920 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-conf\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954968 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-ovn\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.954978 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.454945889 +0000 UTC m=+3.547600959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-host\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrm8c\" (UniqueName: \"kubernetes.io/projected/658fc186-df9a-4f0b-b582-0b727164d3cf-kube-api-access-zrm8c\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955029 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-sys\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955043 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-bin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-kubernetes\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-multus\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955103 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-socket-dir-parent\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-netd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e1842d-4e29-463e-a581-3e102a65c66b-konnectivity-ca\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955151 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cnibin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.957244 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96m5\" (UniqueName: \"kubernetes.io/projected/489888fa-2c2a-4342-a9bc-2dbe32192439-kube-api-access-b96m5\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955214 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-kubelet\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-device-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955273 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-host\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-slash\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysconfig\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gdv\" (UniqueName: \"kubernetes.io/projected/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-kube-api-access-z4gdv\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjjp\" (UniqueName: \"kubernetes.io/projected/717ab396-8f0c-45aa-bd46-6d524b982255-kube-api-access-tkjjp\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-dbus\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955519 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-bin\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-log-socket\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-cni-binary-copy\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.958120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955613 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-log-socket\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovnkube-script-lib\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-slash\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysconfig\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-sysctl-conf\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-serviceca\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955719 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-host-var-lib-cni-multus\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cnibin\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955752 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5958b2d8-31bd-4b8d-8672-024520ce6ef6-dbus\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955771 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-netd\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccxf\" (UniqueName: \"kubernetes.io/projected/55a79f05-3f65-487b-8eed-f9ffd65acf95-kube-api-access-8ccxf\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.954032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-multus-cni-dir\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/489888fa-2c2a-4342-a9bc-2dbe32192439-iptables-alerter-script\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717ab396-8f0c-45aa-bd46-6d524b982255-tmp-dir\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-env-overrides\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-var-lib-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.958964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.955972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-var-lib-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956018 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-node-log\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956046 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-run-openvswitch\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-bin\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-modprobe-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-node-log\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956214 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-host-cni-bin\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956228 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-modprobe-d\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.956994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-etc-tuned\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.957203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-ovn-node-metrics-cert\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.959861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.957829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/658fc186-df9a-4f0b-b582-0b727164d3cf-tmp\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:57.966310 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.966255 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:49:57.966310 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.966282 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:49:57.966310 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.966295 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:57.966516 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:57.966360 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.466343165 +0000 UTC m=+3.558998240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:57.967426 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.967407 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/6a7352a6-b965-48aa-adba-e3fddcdb5fd5-kube-api-access-7hr28\") pod \"multus-rpxdg\" (UID: \"6a7352a6-b965-48aa-adba-e3fddcdb5fd5\") " pod="openshift-multus/multus-rpxdg" Sep 18 17:49:57.968106 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.968089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xnr\" (UniqueName: \"kubernetes.io/projected/2862828c-e9b6-46bf-9a8a-b2a064a41a0a-kube-api-access-s5xnr\") pod \"ovnkube-node-54nft\" (UID: \"2862828c-e9b6-46bf-9a8a-b2a064a41a0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:57.968950 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:57.968926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrm8c\" (UniqueName: \"kubernetes.io/projected/658fc186-df9a-4f0b-b582-0b727164d3cf-kube-api-access-zrm8c\") pod \"tuned-d6n2d\" (UID: \"658fc186-df9a-4f0b-b582-0b727164d3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:58.009335 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.009312 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:49:58.056336 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-system-cni-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.056408 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717ab396-8f0c-45aa-bd46-6d524b982255-hosts-file\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.056408 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98wj\" (UniqueName: \"kubernetes.io/projected/d154cf9d-64bf-413d-a602-5a590e06b22f-kube-api-access-h98wj\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056478 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-system-cni-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.056478 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e1842d-4e29-463e-a581-3e102a65c66b-agent-certs\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:58.056478 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056583 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056422 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717ab396-8f0c-45aa-bd46-6d524b982255-hosts-file\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.056583 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-host\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.056583 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056539 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:58.056583 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056583 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056565 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798dj\" (UniqueName: \"kubernetes.io/projected/5974dc88-f16c-4843-b2a2-9309f55d5ecb-kube-api-access-798dj\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.056754 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-host\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.056754 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056754 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.056657 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:58.056754 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.056716 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.556696056 +0000 UTC m=+3.649351133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:58.056754 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e1842d-4e29-463e-a581-3e102a65c66b-konnectivity-ca\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96m5\" (UniqueName: \"kubernetes.io/projected/489888fa-2c2a-4342-a9bc-2dbe32192439-kube-api-access-b96m5\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-device-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gdv\" (UniqueName: \"kubernetes.io/projected/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-kube-api-access-z4gdv\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056889 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-device-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjjp\" (UniqueName: \"kubernetes.io/projected/717ab396-8f0c-45aa-bd46-6d524b982255-kube-api-access-tkjjp\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056960 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.056978 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.056982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-serviceca\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cnibin\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccxf\" (UniqueName: \"kubernetes.io/projected/55a79f05-3f65-487b-8eed-f9ffd65acf95-kube-api-access-8ccxf\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/489888fa-2c2a-4342-a9bc-2dbe32192439-iptables-alerter-script\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057138 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717ab396-8f0c-45aa-bd46-6d524b982255-tmp-dir\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cnibin\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489888fa-2c2a-4342-a9bc-2dbe32192439-host-slash\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057270 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5974dc88-f16c-4843-b2a2-9309f55d5ecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/489888fa-2c2a-4342-a9bc-2dbe32192439-host-slash\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057340 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d154cf9d-64bf-413d-a602-5a590e06b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057358 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e1842d-4e29-463e-a581-3e102a65c66b-konnectivity-ca\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:58.057994 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717ab396-8f0c-45aa-bd46-6d524b982255-tmp-dir\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.058535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.058535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/489888fa-2c2a-4342-a9bc-2dbe32192439-iptables-alerter-script\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.058535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-serviceca\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.058535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057817 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.058535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.057850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5974dc88-f16c-4843-b2a2-9309f55d5ecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.058776 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.058759 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e1842d-4e29-463e-a581-3e102a65c66b-agent-certs\") pod \"konnectivity-agent-p5t9l\" (UID: \"d6e1842d-4e29-463e-a581-3e102a65c66b\") " pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:58.061103 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.061058 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf4d681d04ec5f43785f7bccdc8c978.slice/crio-11d28956676dcc7945b3d7a96333b88fb52ce761ecfcca696a315de5b854e6ec WatchSource:0}: Error finding container 11d28956676dcc7945b3d7a96333b88fb52ce761ecfcca696a315de5b854e6ec: Status 404 returned error can't find the container with id 11d28956676dcc7945b3d7a96333b88fb52ce761ecfcca696a315de5b854e6ec Sep 18 17:49:58.061523 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.061497 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46296604ae17500791f24b361b85e841.slice/crio-e81292a2d42642fc1d17caa5c7259da9cf824be4586d36ec1a529654034329ad WatchSource:0}: Error finding container e81292a2d42642fc1d17caa5c7259da9cf824be4586d36ec1a529654034329ad: Status 404 returned error can't find the container with id e81292a2d42642fc1d17caa5c7259da9cf824be4586d36ec1a529654034329ad Sep 18 17:49:58.067017 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.066998 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:49:58.067571 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.067547 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96m5\" (UniqueName: \"kubernetes.io/projected/489888fa-2c2a-4342-a9bc-2dbe32192439-kube-api-access-b96m5\") pod \"iptables-alerter-whmld\" (UID: \"489888fa-2c2a-4342-a9bc-2dbe32192439\") " pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.067676 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.067646 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98wj\" (UniqueName: \"kubernetes.io/projected/d154cf9d-64bf-413d-a602-5a590e06b22f-kube-api-access-h98wj\") pod \"aws-ebs-csi-driver-node-xf9xk\" (UID: \"d154cf9d-64bf-413d-a602-5a590e06b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.067745 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.067691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gdv\" (UniqueName: \"kubernetes.io/projected/a88fbff8-83e6-46e6-8126-87ab6c7d9a7b-kube-api-access-z4gdv\") pod \"node-ca-g6r9r\" (UID: \"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b\") " pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.067817 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.067795 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-798dj\" (UniqueName: \"kubernetes.io/projected/5974dc88-f16c-4843-b2a2-9309f55d5ecb-kube-api-access-798dj\") pod \"multus-additional-cni-plugins-d2wmf\" (UID: \"5974dc88-f16c-4843-b2a2-9309f55d5ecb\") " pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.068114 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.068098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjjp\" (UniqueName: \"kubernetes.io/projected/717ab396-8f0c-45aa-bd46-6d524b982255-kube-api-access-tkjjp\") pod \"node-resolver-j6k5v\" (UID: \"717ab396-8f0c-45aa-bd46-6d524b982255\") " pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.068164 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.068149 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccxf\" (UniqueName: \"kubernetes.io/projected/55a79f05-3f65-487b-8eed-f9ffd65acf95-kube-api-access-8ccxf\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:58.156272 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.156219 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:49:58.171495 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.171479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rpxdg" Sep 18 17:49:58.180802 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.180781 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7352a6_b965_48aa_adba_e3fddcdb5fd5.slice/crio-d6f31880021b32aaae9f7dff3e3865f3f3c58bff0637494a7e7765e61d533007 WatchSource:0}: Error finding container d6f31880021b32aaae9f7dff3e3865f3f3c58bff0637494a7e7765e61d533007: Status 404 returned error can't find the container with id d6f31880021b32aaae9f7dff3e3865f3f3c58bff0637494a7e7765e61d533007 Sep 18 17:49:58.184907 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.184890 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:49:58.192572 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.192552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" Sep 18 17:49:58.192761 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.192744 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2862828c_e9b6_46bf_9a8a_b2a064a41a0a.slice/crio-805348d954c33dcd03ac166b7c60c86687d8d3839e3d93b2b539c41f6532675f WatchSource:0}: Error finding container 805348d954c33dcd03ac166b7c60c86687d8d3839e3d93b2b539c41f6532675f: Status 404 returned error can't find the container with id 805348d954c33dcd03ac166b7c60c86687d8d3839e3d93b2b539c41f6532675f Sep 18 17:49:58.200196 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.200168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j6k5v" Sep 18 17:49:58.200747 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.200730 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658fc186_df9a_4f0b_b582_0b727164d3cf.slice/crio-ace161eb7d19c688a2700a183917c077d8313adddeab1564dec52a105aa4ee9e WatchSource:0}: Error finding container ace161eb7d19c688a2700a183917c077d8313adddeab1564dec52a105aa4ee9e: Status 404 returned error can't find the container with id ace161eb7d19c688a2700a183917c077d8313adddeab1564dec52a105aa4ee9e Sep 18 17:49:58.209525 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.209503 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717ab396_8f0c_45aa_bd46_6d524b982255.slice/crio-f243fdce9a5f3233f8e0712d802703a80bffc22d2cae05a2c30fc855def3d6bb WatchSource:0}: Error finding container f243fdce9a5f3233f8e0712d802703a80bffc22d2cae05a2c30fc855def3d6bb: Status 404 returned error can't find the container with id f243fdce9a5f3233f8e0712d802703a80bffc22d2cae05a2c30fc855def3d6bb Sep 18 17:49:58.221168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.221149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g6r9r" Sep 18 17:49:58.229002 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.228984 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88fbff8_83e6_46e6_8126_87ab6c7d9a7b.slice/crio-b9c11d8c6bc0c1a4ba9268d641d6af48b27f0d008dcc5b16269e6784db6d0065 WatchSource:0}: Error finding container b9c11d8c6bc0c1a4ba9268d641d6af48b27f0d008dcc5b16269e6784db6d0065: Status 404 returned error can't find the container with id b9c11d8c6bc0c1a4ba9268d641d6af48b27f0d008dcc5b16269e6784db6d0065 Sep 18 17:49:58.240287 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.240268 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" Sep 18 17:49:58.247384 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.247366 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5974dc88_f16c_4843_b2a2_9309f55d5ecb.slice/crio-8895b5390698a8153c6e7694541120b91ea05e980b8bc48219e97034ee7f1fac WatchSource:0}: Error finding container 8895b5390698a8153c6e7694541120b91ea05e980b8bc48219e97034ee7f1fac: Status 404 returned error can't find the container with id 8895b5390698a8153c6e7694541120b91ea05e980b8bc48219e97034ee7f1fac Sep 18 17:49:58.257164 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.257149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whmld" Sep 18 17:49:58.264633 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.264615 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489888fa_2c2a_4342_a9bc_2dbe32192439.slice/crio-a72e19c9bc35ad6ef5252720cf4c60fd3907cbf2e530c95e4f069ed8fca17b88 WatchSource:0}: Error finding container a72e19c9bc35ad6ef5252720cf4c60fd3907cbf2e530c95e4f069ed8fca17b88: Status 404 returned error can't find the container with id a72e19c9bc35ad6ef5252720cf4c60fd3907cbf2e530c95e4f069ed8fca17b88 Sep 18 17:49:58.275494 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.275479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:49:58.280080 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.280061 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" Sep 18 17:49:58.284755 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.284739 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e1842d_4e29_463e_a581_3e102a65c66b.slice/crio-3d07dd4897d65bfbd3bbbaaa7e3efa54f4c0bc5ca3b3549fa18d54c78e637218 WatchSource:0}: Error finding container 3d07dd4897d65bfbd3bbbaaa7e3efa54f4c0bc5ca3b3549fa18d54c78e637218: Status 404 returned error can't find the container with id 3d07dd4897d65bfbd3bbbaaa7e3efa54f4c0bc5ca3b3549fa18d54c78e637218 Sep 18 17:49:58.288345 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:49:58.288327 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd154cf9d_64bf_413d_a602_5a590e06b22f.slice/crio-0e33ea444f090d097405c9ee0160299129e76ce0ad23dc928b112eae1d268759 WatchSource:0}: Error finding container 0e33ea444f090d097405c9ee0160299129e76ce0ad23dc928b112eae1d268759: Status 404 returned error can't find the container with id 0e33ea444f090d097405c9ee0160299129e76ce0ad23dc928b112eae1d268759 Sep 18 17:49:58.422893 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.422813 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:49:58.460673 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.460646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:58.460801 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.460780 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:58.460839 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.460832 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.460818685 +0000 UTC m=+4.553473755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:58.561949 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.561922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:58.562065 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.561967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:58.562105 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562067 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:58.562105 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562082 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:49:58.562105 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562099 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:49:58.562214 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562110 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:58.562214 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562123 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.562106686 +0000 UTC m=+4.654761764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:58.562214 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.562142 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.562133119 +0000 UTC m=+4.654788201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:58.905210 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.905117 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:44:57 +0000 UTC" deadline="2028-03-19 14:49:19.519922789 +0000 UTC" Sep 18 17:49:58.905210 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.905160 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13148h59m20.61476671s" Sep 18 17:49:58.937396 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.937357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:58.937538 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.937493 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:49:58.939798 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.939775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:58.939909 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:58.939888 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:49:58.958036 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.956326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g6r9r" event={"ID":"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b","Type":"ContainerStarted","Data":"b9c11d8c6bc0c1a4ba9268d641d6af48b27f0d008dcc5b16269e6784db6d0065"} Sep 18 17:49:58.962270 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.962243 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j6k5v" event={"ID":"717ab396-8f0c-45aa-bd46-6d524b982255","Type":"ContainerStarted","Data":"f243fdce9a5f3233f8e0712d802703a80bffc22d2cae05a2c30fc855def3d6bb"} Sep 18 17:49:58.975161 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.975130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rpxdg" event={"ID":"6a7352a6-b965-48aa-adba-e3fddcdb5fd5","Type":"ContainerStarted","Data":"d6f31880021b32aaae9f7dff3e3865f3f3c58bff0637494a7e7765e61d533007"} Sep 18 17:49:58.980690 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.980664 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" event={"ID":"edf4d681d04ec5f43785f7bccdc8c978","Type":"ContainerStarted","Data":"11d28956676dcc7945b3d7a96333b88fb52ce761ecfcca696a315de5b854e6ec"} Sep 18 17:49:58.994171 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:58.994044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whmld" event={"ID":"489888fa-2c2a-4342-a9bc-2dbe32192439","Type":"ContainerStarted","Data":"a72e19c9bc35ad6ef5252720cf4c60fd3907cbf2e530c95e4f069ed8fca17b88"} Sep 18 17:49:59.002225 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.002125 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerStarted","Data":"8895b5390698a8153c6e7694541120b91ea05e980b8bc48219e97034ee7f1fac"} Sep 18 17:49:59.007498 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.007463 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" event={"ID":"658fc186-df9a-4f0b-b582-0b727164d3cf","Type":"ContainerStarted","Data":"ace161eb7d19c688a2700a183917c077d8313adddeab1564dec52a105aa4ee9e"} Sep 18 17:49:59.033031 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.032922 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"805348d954c33dcd03ac166b7c60c86687d8d3839e3d93b2b539c41f6532675f"} Sep 18 17:49:59.044351 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.044284 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" event={"ID":"46296604ae17500791f24b361b85e841","Type":"ContainerStarted","Data":"e81292a2d42642fc1d17caa5c7259da9cf824be4586d36ec1a529654034329ad"} Sep 18 17:49:59.057177 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.057150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" event={"ID":"d154cf9d-64bf-413d-a602-5a590e06b22f","Type":"ContainerStarted","Data":"0e33ea444f090d097405c9ee0160299129e76ce0ad23dc928b112eae1d268759"} Sep 18 17:49:59.058935 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.058911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5t9l" event={"ID":"d6e1842d-4e29-463e-a581-3e102a65c66b","Type":"ContainerStarted","Data":"3d07dd4897d65bfbd3bbbaaa7e3efa54f4c0bc5ca3b3549fa18d54c78e637218"} Sep 18 17:49:59.469151 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.469111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:59.469332 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.469300 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:59.469412 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.469364 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.469346999 +0000 UTC m=+6.562002087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:49:59.570546 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.570505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:49:59.570720 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.570573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:49:59.570720 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.570688 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:59.570890 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.570751 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.570732018 +0000 UTC m=+6.663387104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:49:59.571200 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.571157 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:49:59.571200 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.571194 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:49:59.571340 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.571209 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:59.571340 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.571257 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.571241998 +0000 UTC m=+6.663897077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:49:59.905939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.905621 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:44:57 +0000 UTC" deadline="2028-04-14 23:02:21.626685124 +0000 UTC" Sep 18 17:49:59.905939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.905883 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13781h12m21.720807114s" Sep 18 17:49:59.936059 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:49:59.936023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:49:59.936249 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:49:59.936144 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:00.938684 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:00.938650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:00.939103 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:00.938779 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:00.939174 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:00.939144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:00.939285 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:00.939261 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:01.486996 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:01.486953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:01.487173 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.487107 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:01.487173 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.487168 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.487150823 +0000 UTC m=+10.579805888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:01.588046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:01.588119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588292 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588312 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588324 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588405 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.588364349 +0000 UTC m=+10.681019430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588790 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:01.588880 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.588844 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.588827695 +0000 UTC m=+10.681482781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:01.936902 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:01.936376 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:01.936902 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:01.936502 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:02.937990 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:02.936980 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:02.937990 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:02.937106 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:02.937990 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:02.937589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:02.937990 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:02.937694 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:03.935862 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:03.935829 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:03.936035 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:03.935975 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:04.936162 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:04.936124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:04.936561 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:04.936281 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:04.936561 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:04.936347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:04.936561 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:04.936518 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:05.518114 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:05.518079 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:05.518301 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.518179 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:05.518301 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.518273 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.518252606 +0000 UTC m=+18.610907696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:05.619112 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:05.619069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:05.619278 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:05.619146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:05.619339 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619283 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:05.619391 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619345 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.619326543 +0000 UTC m=+18.711981633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:05.619514 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619488 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:50:05.619585 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619522 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:50:05.619585 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619535 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:05.619699 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.619588 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.619571379 +0000 UTC m=+18.712226465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:05.936400 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:05.936325 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:05.936836 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:05.936455 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:06.936981 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:06.936953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:06.937442 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:06.937033 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:06.937442 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:06.937042 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:06.937442 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:06.937163 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:07.936153 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:07.936118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:07.936344 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:07.936237 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:08.936480 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:08.936441 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:08.936957 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:08.936449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:08.936957 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:08.936573 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:08.936957 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:08.936678 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:09.936549 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:09.936463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:09.936975 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:09.936586 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:10.936173 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:10.936138 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:10.936425 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:10.936284 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:10.936425 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:10.936343 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:10.936528 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:10.936466 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:11.936055 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:11.936021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:11.936475 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:11.936153 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:12.936196 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:12.936146 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:12.936616 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:12.936309 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:12.936616 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:12.936365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:12.936616 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:12.936497 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:13.574473 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:13.574438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:13.574666 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.574627 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:13.574720 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.574705 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.574684599 +0000 UTC m=+34.667339670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:13.675103 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:13.675071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:13.675293 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:13.675127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:13.675293 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675256 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:13.675410 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675292 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:50:13.675410 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675310 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.675297106 +0000 UTC m=+34.767952171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:13.675410 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675316 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:50:13.675410 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675330 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:13.675410 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.675382 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.675365878 +0000 UTC m=+34.768020961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:13.936726 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:13.936657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:13.937211 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:13.936782 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:14.936024 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:14.935987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:14.936225 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:14.936103 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:14.936225 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:14.936164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:14.936341 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:14.936296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:15.936528 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:15.936491 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:15.936983 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:15.936611 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:16.936240 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:16.936213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:16.936390 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:16.936242 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:16.936390 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:16.936299 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:16.936502 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:16.936478 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:17.936060 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:17.935797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:17.936836 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:17.936103 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:18.107220 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.107171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" event={"ID":"658fc186-df9a-4f0b-b582-0b727164d3cf","Type":"ContainerStarted","Data":"fa53af1641fe09e1b9f88cf7f65949493563c48eb2712d019b0b5d788b624c82"} Sep 18 17:50:18.109334 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.109313 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"b043b3fad021eee4efabca2fa77f0f2503b682d17188288acd317e828ba5776a"} Sep 18 17:50:18.109421 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.109369 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"ada6958ec310d052b49818d7f050fc0b92d756f159b81dccbca898a90d948a67"} Sep 18 17:50:18.109421 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.109410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"c147389a16d3fcf9391ca84774c850bb2efe724ab1f13685ee54ae5856ed14b6"} Sep 18 17:50:18.110661 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.110641 2570 generic.go:356] "Generic (PLEG): container finished" podID="46296604ae17500791f24b361b85e841" containerID="5d69c25348f4d16b82fe8d5813509cbd6b719ed79f1176544cd0fc1763db9041" exitCode=0 Sep 18 17:50:18.110744 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.110702 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" event={"ID":"46296604ae17500791f24b361b85e841","Type":"ContainerDied","Data":"5d69c25348f4d16b82fe8d5813509cbd6b719ed79f1176544cd0fc1763db9041"} Sep 18 17:50:18.112058 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.112033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" event={"ID":"d154cf9d-64bf-413d-a602-5a590e06b22f","Type":"ContainerStarted","Data":"3b512464d2bef2f98d8e50ad976da426f6beaddef469a35f721c4956574b7a48"} Sep 18 17:50:18.113391 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.113369 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5t9l" event={"ID":"d6e1842d-4e29-463e-a581-3e102a65c66b","Type":"ContainerStarted","Data":"d988b10c37e7fc4f00053b877f71d772b00dd5b13dc4f199b4b2b54fd9b9cb1f"} Sep 18 17:50:18.114810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.114792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g6r9r" event={"ID":"a88fbff8-83e6-46e6-8126-87ab6c7d9a7b","Type":"ContainerStarted","Data":"ff8438fa4a6f0613ef12133088036d7f272d1f40464776c1ffd94222ec004573"} Sep 18 17:50:18.116090 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.116063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j6k5v" event={"ID":"717ab396-8f0c-45aa-bd46-6d524b982255","Type":"ContainerStarted","Data":"14de33820b524f75988ee137423935414242a8bf12ee8146c8a328aa2ce3e93c"} Sep 18 17:50:18.117349 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.117323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rpxdg" event={"ID":"6a7352a6-b965-48aa-adba-e3fddcdb5fd5","Type":"ContainerStarted","Data":"ee9bfed0fce76aca31d7e09740f8170cb8af5df50e62dccc56aeb1aa9d37811d"} Sep 18 17:50:18.121433 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.121409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" event={"ID":"edf4d681d04ec5f43785f7bccdc8c978","Type":"ContainerStarted","Data":"6a9d409af9b9712f55ee08395ad9a403dbfb2efefcd57a688168732e0b1b959d"} Sep 18 17:50:18.122776 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.122754 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="ea7287c5141ee125bed63e858919f51895b3333f8e5c0071f87dd13b1de40ae2" exitCode=0 Sep 18 17:50:18.122841 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.122787 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"ea7287c5141ee125bed63e858919f51895b3333f8e5c0071f87dd13b1de40ae2"} Sep 18 17:50:18.151907 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.151860 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d6n2d" podStartSLOduration=2.163338304 podStartE2EDuration="21.151849609s" totalImagesPullingTime="18.988511305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.151594359 +0000 UTC m=+23.244249471" watchObservedRunningTime="2026-09-18 17:50:18.151849609 +0000 UTC m=+23.244504695" Sep 18 17:50:18.237532 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.237496 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p5t9l" podStartSLOduration=2.33459006 podStartE2EDuration="21.237483409s" totalImagesPullingTime="18.902893349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.202054533 +0000 UTC m=+23.294709652" watchObservedRunningTime="2026-09-18 17:50:18.237483409 +0000 UTC m=+23.330138495" Sep 18 17:50:18.237634 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.237617 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rpxdg" podStartSLOduration=2.223558991 podStartE2EDuration="21.237611327s" totalImagesPullingTime="19.014052336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.235260714 +0000 UTC m=+23.327915822" watchObservedRunningTime="2026-09-18 17:50:18.237611327 +0000 UTC m=+23.330266414" Sep 18 17:50:18.262337 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.262300 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j6k5v" podStartSLOduration=2.282807623 podStartE2EDuration="21.262286008s" totalImagesPullingTime="18.979478385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.261880377 +0000 UTC m=+23.354535485" watchObservedRunningTime="2026-09-18 17:50:18.262286008 +0000 UTC m=+23.354941095" Sep 18 17:50:18.329300 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.329240 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g6r9r" podStartSLOduration=9.94857914 podStartE2EDuration="21.32922433s" totalImagesPullingTime="11.38064519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.298313678 +0000 UTC m=+23.390968767" watchObservedRunningTime="2026-09-18 17:50:18.32922433 +0000 UTC m=+23.421879416" Sep 18 17:50:18.361475 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.361431 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-72.ec2.internal" podStartSLOduration=21.361412413 podStartE2EDuration="21.361412413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.361122683 +0000 UTC m=+23.453777794" watchObservedRunningTime="2026-09-18 17:50:18.361412413 +0000 UTC m=+23.454067503" Sep 18 17:50:18.935954 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.935926 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:18.936103 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:18.936039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:18.936430 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.936109 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:18.936430 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:18.936208 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:18.971493 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:18.971449 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 17:50:19.126145 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.126068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whmld" event={"ID":"489888fa-2c2a-4342-a9bc-2dbe32192439","Type":"ContainerStarted","Data":"a186d38de3d34d46df00765c0bf41cdd0430c5bc5938223c6c3baa756ff56991"} Sep 18 17:50:19.129049 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.129026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"4bb50428b05a2344cae96f62718f20abd7922f4bcdec5db692db61de90ab9f95"} Sep 18 17:50:19.129173 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.129097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"5b089d9202d3d74f454391d442cf2f702dbccffc80d251161b96e889e823e1ba"} Sep 18 17:50:19.129173 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.129148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"7fa858d29d122430bff431b04a48d8eb758b67fb4a23f86290c615822db2596e"} Sep 18 17:50:19.130896 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.130873 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" event={"ID":"46296604ae17500791f24b361b85e841","Type":"ContainerStarted","Data":"cf4a4f53a396e32a79de3c82b11f9d02af0e91158bbdb99e3f56055d81a60461"} Sep 18 17:50:19.132453 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.132431 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" event={"ID":"d154cf9d-64bf-413d-a602-5a590e06b22f","Type":"ContainerStarted","Data":"1694e84931cb622433ebfc654945963ec592434dfaad1462c516dc3af304b093"} Sep 18 17:50:19.165681 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.165644 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-72.ec2.internal" podStartSLOduration=22.165632315 podStartE2EDuration="22.165632315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:19.165451494 +0000 UTC m=+24.258106607" watchObservedRunningTime="2026-09-18 17:50:19.165632315 +0000 UTC m=+24.258287402" Sep 18 17:50:19.166014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.165990 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-whmld" podStartSLOduration=3.241485357 podStartE2EDuration="22.165980333s" totalImagesPullingTime="18.924494976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:19.144931619 +0000 UTC m=+24.237586738" watchObservedRunningTime="2026-09-18 17:50:19.165980333 +0000 UTC m=+24.258635419" Sep 18 17:50:19.545172 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.545099 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:50:19.545852 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.545835 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:50:19.936091 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.935950 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T17:50:18.971467392Z","UUID":"12907071-7e97-4c8f-9b4c-352b21755a92","Handler":null,"Name":"","Endpoint":""} Sep 18 17:50:19.936091 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.936033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:19.936539 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:19.936130 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:19.937539 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.937522 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 17:50:19.937662 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:19.937545 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 17:50:20.134723 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:20.134697 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:50:20.135196 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:20.135167 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p5t9l" Sep 18 17:50:20.936502 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:20.936343 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:20.936877 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:20.936351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:20.936877 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:20.936573 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:20.936877 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:20.936645 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:21.139316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:21.139285 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"1ceb8d53bd2f14f3678df5c548b31d850f7c832132000f62371f26e1fa7064d5"} Sep 18 17:50:21.141172 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:21.141147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" event={"ID":"d154cf9d-64bf-413d-a602-5a590e06b22f","Type":"ContainerStarted","Data":"603e2d28aaa05fee3908bb8f310b35cecb00eedb5cffa667a1fe94b4c5381c92"} Sep 18 17:50:21.160126 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:21.160088 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xf9xk" podStartSLOduration=3.148211806 podStartE2EDuration="24.160075157s" totalImagesPullingTime="21.011863351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:21.159977666 +0000 UTC m=+26.252632969" watchObservedRunningTime="2026-09-18 17:50:21.160075157 +0000 UTC m=+26.252730244" Sep 18 17:50:21.935880 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:21.935845 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:21.936068 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:21.935952 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:22.936802 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:22.936766 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:22.937471 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:22.936897 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:22.937471 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:22.936947 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:22.937471 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:22.937051 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:23.935889 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:23.935711 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:23.936041 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:23.935973 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:24.149303 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.149268 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="3d63b01ff3e6c49f9c387533b4f16b78324045af47b23510c582410386f1a6a1" exitCode=0 Sep 18 17:50:24.149884 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.149349 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"3d63b01ff3e6c49f9c387533b4f16b78324045af47b23510c582410386f1a6a1"} Sep 18 17:50:24.153173 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.153151 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" event={"ID":"2862828c-e9b6-46bf-9a8a-b2a064a41a0a","Type":"ContainerStarted","Data":"7e3672593edb49c8d62e2135d64088d854afdc8a1f8aba7c9c9e808c175a577c"} Sep 18 17:50:24.153515 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.153496 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:24.168168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.168149 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:24.224341 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.224270 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" podStartSLOduration=7.736572844 podStartE2EDuration="27.22425047s" totalImagesPullingTime="19.487677626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:24.222528174 +0000 UTC m=+29.315183282" watchObservedRunningTime="2026-09-18 17:50:24.22425047 +0000 UTC m=+29.316905750" Sep 18 17:50:24.936115 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.936087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:24.936304 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:24.936087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:24.936304 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:24.936215 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:24.936304 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:24.936270 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:25.067618 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.067591 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9vfvg"] Sep 18 17:50:25.071736 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.071712 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrn47"] Sep 18 17:50:25.072395 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.072374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58kvg"] Sep 18 17:50:25.072501 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.072484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:25.072595 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:25.072577 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:25.157498 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.157469 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="d92b838f8a91d6fd742bbbf6f954e19a49d69c1bab6c9a0a06dd8b833d656191" exitCode=0 Sep 18 17:50:25.157828 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.157568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"d92b838f8a91d6fd742bbbf6f954e19a49d69c1bab6c9a0a06dd8b833d656191"} Sep 18 17:50:25.157828 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.157606 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:25.157828 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.157652 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:25.157828 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:25.157811 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:25.158014 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:25.157867 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:25.158615 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.158423 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:25.158615 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.158486 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:25.172638 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:25.172622 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:26.161763 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:26.161590 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="17083452937db8bdb494f011a959011395ef64427da4cce88dd40036ebf83f3b" exitCode=0 Sep 18 17:50:26.162208 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:26.161657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"17083452937db8bdb494f011a959011395ef64427da4cce88dd40036ebf83f3b"} Sep 18 17:50:26.937063 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:26.937034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:26.937241 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:26.937103 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:26.937241 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:26.937125 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:26.937241 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:26.937201 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:26.937241 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:26.937239 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:26.937458 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:26.937314 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:28.936648 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:28.936615 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:28.937156 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:28.936619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:28.937156 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:28.936731 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vfvg" podUID="bf3b5c20-8100-4365-b3d6-a98996cb658b" Sep 18 17:50:28.937156 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:28.936842 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrn47" podUID="55a79f05-3f65-487b-8eed-f9ffd65acf95" Sep 18 17:50:28.937156 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:28.936616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:28.937156 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:28.936934 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58kvg" podUID="5958b2d8-31bd-4b8d-8672-024520ce6ef6" Sep 18 17:50:29.588345 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:29.588312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:29.588521 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.588447 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:29.588521 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.588502 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret podName:5958b2d8-31bd-4b8d-8672-024520ce6ef6 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.588488721 +0000 UTC m=+66.681143785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret") pod "global-pull-secret-syncer-58kvg" (UID: "5958b2d8-31bd-4b8d-8672-024520ce6ef6") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:50:29.689546 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:29.689518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:29.689725 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:29.689573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:29.689725 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689692 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:29.689725 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689704 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:50:29.689725 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689723 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:50:29.689926 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689733 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kkk7b for pod openshift-network-diagnostics/network-check-target-9vfvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:29.689926 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689768 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs podName:55a79f05-3f65-487b-8eed-f9ffd65acf95 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.689746765 +0000 UTC m=+66.782401847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs") pod "network-metrics-daemon-vrn47" (UID: "55a79f05-3f65-487b-8eed-f9ffd65acf95") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:50:29.689926 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:29.689788 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b podName:bf3b5c20-8100-4365-b3d6-a98996cb658b nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.68977803 +0000 UTC m=+66.782433099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkk7b" (UniqueName: "kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b") pod "network-check-target-9vfvg" (UID: "bf3b5c20-8100-4365-b3d6-a98996cb658b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:50:30.391728 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.391651 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-72.ec2.internal" event="NodeReady" Sep 18 17:50:30.392392 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.391798 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 17:50:30.428122 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.428089 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:50:30.486484 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.486446 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5ll45"] Sep 18 17:50:30.486655 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.486585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.489712 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.489672 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 17:50:30.489836 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.489731 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 17:50:30.489949 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.489933 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xssnq\"" Sep 18 17:50:30.489998 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.489933 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 17:50:30.497216 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.497196 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 17:50:30.501746 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.501728 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gmlnt"] Sep 18 17:50:30.501844 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.501836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.504220 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.504202 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 17:50:30.504458 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.504438 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s9cmf\"" Sep 18 17:50:30.504458 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.504449 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 17:50:30.517814 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.517792 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:50:30.517910 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.517829 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ll45"] Sep 18 17:50:30.517910 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.517894 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.517986 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.517927 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmlnt"] Sep 18 17:50:30.522233 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.522216 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhc9g\"" Sep 18 17:50:30.522337 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.522250 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 17:50:30.522416 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.522393 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 17:50:30.522738 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.522722 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 17:50:30.595387 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595387 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdj8\" (UniqueName: \"kubernetes.io/projected/683adc43-c28a-4c5f-a783-88a97ff26e30-kube-api-access-zhdj8\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwvk\" (UniqueName: \"kubernetes.io/projected/9172cf61-f370-4720-9191-b5ea2f68d55e-kube-api-access-nxwvk\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4prm\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9172cf61-f370-4720-9191-b5ea2f68d55e-tmp-dir\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595791 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9172cf61-f370-4720-9191-b5ea2f68d55e-config-volume\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683adc43-c28a-4c5f-a783-88a97ff26e30-cert\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.595890 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.595866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.696658 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9172cf61-f370-4720-9191-b5ea2f68d55e-tmp-dir\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.696658 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9172cf61-f370-4720-9191-b5ea2f68d55e-config-volume\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683adc43-c28a-4c5f-a783-88a97ff26e30-cert\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:30.696769 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:30.696787 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:30.696834 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:30.696881 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:30.696859 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.196841432 +0000 UTC m=+36.289496520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:30.696921 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.196908721 +0000 UTC m=+36.289563795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.696999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdj8\" (UniqueName: \"kubernetes.io/projected/683adc43-c28a-4c5f-a783-88a97ff26e30-kube-api-access-zhdj8\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.697058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.697086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.697214 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.697052 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9172cf61-f370-4720-9191-b5ea2f68d55e-tmp-dir\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.697655 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.697605 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwvk\" (UniqueName: \"kubernetes.io/projected/9172cf61-f370-4720-9191-b5ea2f68d55e-kube-api-access-nxwvk\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.697868 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.697834 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9172cf61-f370-4720-9191-b5ea2f68d55e-config-volume\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.698176 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.698114 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.698319 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.698212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.698319 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.698266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.698319 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.698294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4prm\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.698801 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.698530 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.702070 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.702046 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.702168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.702150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683adc43-c28a-4c5f-a783-88a97ff26e30-cert\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.702711 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.702681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.702711 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.702702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.706895 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.706874 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdj8\" (UniqueName: \"kubernetes.io/projected/683adc43-c28a-4c5f-a783-88a97ff26e30-kube-api-access-zhdj8\") pod \"ingress-canary-gmlnt\" (UID: \"683adc43-c28a-4c5f-a783-88a97ff26e30\") " pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.707040 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.707016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwvk\" (UniqueName: \"kubernetes.io/projected/9172cf61-f370-4720-9191-b5ea2f68d55e-kube-api-access-nxwvk\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:30.707957 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.707934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4prm\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.708716 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.708692 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:30.826811 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.826774 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmlnt" Sep 18 17:50:30.936225 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.936175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:50:30.936399 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.936175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:50:30.936585 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.936199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:50:30.939600 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939573 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 17:50:30.939699 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939654 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sdsc5\"" Sep 18 17:50:30.939810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939795 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 17:50:30.939864 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939850 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5l4c\"" Sep 18 17:50:30.939897 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 17:50:30.939931 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:30.939915 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 17:50:31.203451 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:31.203239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:31.203451 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:31.203430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:31.203606 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:31.203382 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:31.203606 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:31.203478 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:31.203606 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:31.203528 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:32.203512261 +0000 UTC m=+37.296167326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:31.203606 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:31.203541 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:31.203735 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:31.203601 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:50:32.203568998 +0000 UTC m=+37.296224062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:31.331095 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:31.331068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmlnt"] Sep 18 17:50:31.334444 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:50:31.334421 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683adc43_c28a_4c5f_a783_88a97ff26e30.slice/crio-3e7d75ca92af38c4efe08eee77e4a4b5c91092560e7edfafddee5d9d1ce6db17 WatchSource:0}: Error finding container 3e7d75ca92af38c4efe08eee77e4a4b5c91092560e7edfafddee5d9d1ce6db17: Status 404 returned error can't find the container with id 3e7d75ca92af38c4efe08eee77e4a4b5c91092560e7edfafddee5d9d1ce6db17 Sep 18 17:50:32.181559 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:32.181528 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="161e45c3a8e25a80167556c2da349aff6e630b1fe0f02afc438c4920b9bb9795" exitCode=0 Sep 18 17:50:32.182145 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:32.181609 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"161e45c3a8e25a80167556c2da349aff6e630b1fe0f02afc438c4920b9bb9795"} Sep 18 17:50:32.182703 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:32.182684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmlnt" event={"ID":"683adc43-c28a-4c5f-a783-88a97ff26e30","Type":"ContainerStarted","Data":"3e7d75ca92af38c4efe08eee77e4a4b5c91092560e7edfafddee5d9d1ce6db17"} Sep 18 17:50:32.212630 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:32.212600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:32.212721 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:32.212674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:32.212838 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:32.212726 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:32.212838 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:32.212744 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:32.212838 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:32.212770 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:32.212838 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:32.212801 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:34.212783039 +0000 UTC m=+39.305438120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:32.212838 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:32.212819 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:50:34.212809967 +0000 UTC m=+39.305465038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:33.189083 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:33.189033 2570 generic.go:356] "Generic (PLEG): container finished" podID="5974dc88-f16c-4843-b2a2-9309f55d5ecb" containerID="89acd57d9882ae92840c0ad5ea686b443931603d0ff854bebef7b88dc01cd71a" exitCode=0 Sep 18 17:50:33.189641 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:33.189114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerDied","Data":"89acd57d9882ae92840c0ad5ea686b443931603d0ff854bebef7b88dc01cd71a"} Sep 18 17:50:34.194496 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:34.194462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" event={"ID":"5974dc88-f16c-4843-b2a2-9309f55d5ecb","Type":"ContainerStarted","Data":"df0daf2db6614ddd4e3182a87fe1d77414530f23cd5eee35f5f97b1cd778f8d0"} Sep 18 17:50:34.226592 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:34.226562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:34.226732 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:34.226713 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:34.226732 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:34.226732 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:34.226985 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:34.226727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:34.226985 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:34.226790 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:38.226770205 +0000 UTC m=+43.319425287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:34.226985 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:34.226834 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:34.226985 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:34.226891 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:50:38.22687645 +0000 UTC m=+43.319531515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:34.227505 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:34.227470 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d2wmf" podStartSLOduration=6.385615981 podStartE2EDuration="37.227459092s" totalImagesPullingTime="30.841843111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:34.226378606 +0000 UTC m=+39.319033714" watchObservedRunningTime="2026-09-18 17:50:34.227459092 +0000 UTC m=+39.320114178" Sep 18 17:50:35.198723 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:35.198538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmlnt" event={"ID":"683adc43-c28a-4c5f-a783-88a97ff26e30","Type":"ContainerStarted","Data":"3caa3c5032935dc51cb1a72bef0be337866754d68895e9beb08a5abf844b71be"} Sep 18 17:50:35.220495 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:35.220453 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gmlnt" podStartSLOduration=2.237062821 podStartE2EDuration="5.220439688s" totalImagesPullingTime="2.983376867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:35.21972945 +0000 UTC m=+40.312384550" watchObservedRunningTime="2026-09-18 17:50:35.220439688 +0000 UTC m=+40.313094775" Sep 18 17:50:38.256256 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:38.256218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:38.256275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:38.256360 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:38.256388 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:38.256409 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:50:46.256394566 +0000 UTC m=+51.349049635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:38.256409 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:38.256739 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:38.256461 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:46.256445883 +0000 UTC m=+51.349100966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:41.459724 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.459687 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5ds66"] Sep 18 17:50:41.492487 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.492456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ds66"] Sep 18 17:50:41.492624 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.492556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.495296 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.495271 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 17:50:41.495426 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.495271 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vmk8k\"" Sep 18 17:50:41.495426 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.495381 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 17:50:41.495868 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.495854 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 17:50:41.495868 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.495862 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 17:50:41.578407 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.578387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.578519 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.578413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-data-volume\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.578519 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.578454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-crio-socket\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.578519 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.578485 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.578519 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.578515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftstg\" (UniqueName: \"kubernetes.io/projected/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-api-access-ftstg\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679374 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-crio-socket\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679476 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679476 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679398 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftstg\" (UniqueName: \"kubernetes.io/projected/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-api-access-ftstg\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679476 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679621 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-data-volume\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679621 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:41.679510 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:50:41.679621 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679539 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-crio-socket\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.679621 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:41.679575 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:50:41.679621 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:41.679592 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:42.179570268 +0000 UTC m=+47.272225347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : secret "insights-runtime-extractor-tls" not found Sep 18 17:50:41.679816 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:41.679631 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:42.179618363 +0000 UTC m=+47.272273452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : configmap "kube-rbac-proxy" not found Sep 18 17:50:41.679816 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.679781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-data-volume\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:41.713816 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:41.713772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftstg\" (UniqueName: \"kubernetes.io/projected/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-api-access-ftstg\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:42.183279 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:42.183254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:42.183447 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:42.183322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:42.183447 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:42.183411 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:50:42.183447 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:42.183417 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:50:42.183581 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:42.183466 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:43.183451616 +0000 UTC m=+48.276106684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : configmap "kube-rbac-proxy" not found Sep 18 17:50:42.183581 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:42.183493 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:43.183474317 +0000 UTC m=+48.276129401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : secret "insights-runtime-extractor-tls" not found Sep 18 17:50:43.191333 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:43.191289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:43.191762 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:43.191384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:43.191762 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:43.191413 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:50:43.191762 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:43.191481 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.191464786 +0000 UTC m=+50.284119850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : configmap "kube-rbac-proxy" not found Sep 18 17:50:43.191762 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:43.191523 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:50:43.191762 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:43.191581 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.19156541 +0000 UTC m=+50.284220477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : secret "insights-runtime-extractor-tls" not found Sep 18 17:50:44.009438 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:44.009411 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j6k5v_717ab396-8f0c-45aa-bd46-6d524b982255/dns-node-resolver/0.log" Sep 18 17:50:45.009822 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:45.009792 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g6r9r_a88fbff8-83e6-46e6-8126-87ab6c7d9a7b/node-ca/0.log" Sep 18 17:50:45.205477 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:45.205440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:45.205627 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:45.205531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:45.205627 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:45.205565 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:50:45.205737 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:45.205633 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:49.205616802 +0000 UTC m=+54.298271887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : configmap "kube-rbac-proxy" not found Sep 18 17:50:45.205737 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:45.205658 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:50:45.205737 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:45.205706 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:49.205693444 +0000 UTC m=+54.298348508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : secret "insights-runtime-extractor-tls" not found Sep 18 17:50:45.829422 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:45.829393 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmlnt_683adc43-c28a-4c5f-a783-88a97ff26e30/serve-healthcheck-canary/0.log" Sep 18 17:50:46.313827 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:46.313796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:46.313878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:46.313944 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:46.313975 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:46.313988 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657d898764-rxf7r: secret "image-registry-tls" not found Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:46.314005 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls podName:9172cf61-f370-4720-9191-b5ea2f68d55e nodeName:}" failed. No retries permitted until 2026-09-18 17:51:02.313991407 +0000 UTC m=+67.406646473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls") pod "dns-default-5ll45" (UID: "9172cf61-f370-4720-9191-b5ea2f68d55e") : secret "dns-default-metrics-tls" not found Sep 18 17:50:46.314206 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:46.314027 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls podName:29657b39-949d-4569-805a-6122d7c864f0 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:02.314016475 +0000 UTC m=+67.406671540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls") pod "image-registry-657d898764-rxf7r" (UID: "29657b39-949d-4569-805a-6122d7c864f0") : secret "image-registry-tls" not found Sep 18 17:50:49.238951 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:49.238917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:49.239320 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:49.239054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:49.239320 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:49.239055 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:50:49.239320 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:50:49.239122 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls podName:53ad7cfa-75ac-4f0a-b524-84def11fb4ce nodeName:}" failed. No retries permitted until 2026-09-18 17:50:57.239108103 +0000 UTC m=+62.331763169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ds66" (UID: "53ad7cfa-75ac-4f0a-b524-84def11fb4ce") : secret "insights-runtime-extractor-tls" not found Sep 18 17:50:49.239509 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:49.239490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:57.173716 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.173690 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nft" Sep 18 17:50:57.297843 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.297795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:57.300175 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.300143 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ad7cfa-75ac-4f0a-b524-84def11fb4ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ds66\" (UID: \"53ad7cfa-75ac-4f0a-b524-84def11fb4ce\") " pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:57.404815 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.404785 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vmk8k\"" Sep 18 17:50:57.412298 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.412281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ds66" Sep 18 17:50:57.687430 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:57.686357 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ds66"] Sep 18 17:50:57.691278 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:50:57.691247 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ad7cfa_75ac_4f0a_b524_84def11fb4ce.slice/crio-ff8263c78aad4f00f1ad69df95cdf955e6c498a2ce6d099e497e00a4f698042d WatchSource:0}: Error finding container ff8263c78aad4f00f1ad69df95cdf955e6c498a2ce6d099e497e00a4f698042d: Status 404 returned error can't find the container with id ff8263c78aad4f00f1ad69df95cdf955e6c498a2ce6d099e497e00a4f698042d Sep 18 17:50:58.252604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:58.252571 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ds66" event={"ID":"53ad7cfa-75ac-4f0a-b524-84def11fb4ce","Type":"ContainerStarted","Data":"95944591c66d993e4a9eba3aa1b569c361f82f662ca38e99f136f277b11af1e5"} Sep 18 17:50:58.253054 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:58.252629 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ds66" event={"ID":"53ad7cfa-75ac-4f0a-b524-84def11fb4ce","Type":"ContainerStarted","Data":"ff8263c78aad4f00f1ad69df95cdf955e6c498a2ce6d099e497e00a4f698042d"} Sep 18 17:50:59.258088 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:50:59.258047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ds66" event={"ID":"53ad7cfa-75ac-4f0a-b524-84def11fb4ce","Type":"ContainerStarted","Data":"946dbbf399e0efe8d70e8050ae661d32f570747ece57b70d9270da5954b61d4d"} Sep 18 17:51:00.263111 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:00.263079 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ds66" event={"ID":"53ad7cfa-75ac-4f0a-b524-84def11fb4ce","Type":"ContainerStarted","Data":"426a7b6e2d0914847ec55d91f086a3c27246e4a4b7cc250205f63fe408091e8b"} Sep 18 17:51:01.631271 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.631227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:51:01.634812 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.634789 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 17:51:01.644053 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.644026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5958b2d8-31bd-4b8d-8672-024520ce6ef6-original-pull-secret\") pod \"global-pull-secret-syncer-58kvg\" (UID: \"5958b2d8-31bd-4b8d-8672-024520ce6ef6\") " pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:51:01.732000 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.731966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:51:01.732152 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.732019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:51:01.734887 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.734870 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 17:51:01.734887 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.734879 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 17:51:01.745231 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.745203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a79f05-3f65-487b-8eed-f9ffd65acf95-metrics-certs\") pod \"network-metrics-daemon-vrn47\" (UID: \"55a79f05-3f65-487b-8eed-f9ffd65acf95\") " pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:51:01.745772 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.745753 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 17:51:01.755975 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.755955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk7b\" (UniqueName: \"kubernetes.io/projected/bf3b5c20-8100-4365-b3d6-a98996cb658b-kube-api-access-kkk7b\") pod \"network-check-target-9vfvg\" (UID: \"bf3b5c20-8100-4365-b3d6-a98996cb658b\") " pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:51:01.850923 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.850898 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5l4c\"" Sep 18 17:51:01.855852 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.855837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58kvg" Sep 18 17:51:01.858963 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.858946 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:51:01.864542 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.864522 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sdsc5\"" Sep 18 17:51:01.872685 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:01.872671 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrn47" Sep 18 17:51:02.149642 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.149553 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5ds66" podStartSLOduration=19.22606866 podStartE2EDuration="21.149537148s" totalImagesPullingTime="1.923468488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:50:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:00.287308375 +0000 UTC m=+65.379963495" watchObservedRunningTime="2026-09-18 17:51:02.149537148 +0000 UTC m=+67.242192234" Sep 18 17:51:02.149825 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.149812 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58kvg"] Sep 18 17:51:02.152561 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:02.152536 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5958b2d8_31bd_4b8d_8672_024520ce6ef6.slice/crio-53f59cc839152bb41c11bf8840716d6b5e4781a8110a9973d59daf510c8052d7 WatchSource:0}: Error finding container 53f59cc839152bb41c11bf8840716d6b5e4781a8110a9973d59daf510c8052d7: Status 404 returned error can't find the container with id 53f59cc839152bb41c11bf8840716d6b5e4781a8110a9973d59daf510c8052d7 Sep 18 17:51:02.179212 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.179176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9vfvg"] Sep 18 17:51:02.182387 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:02.182360 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3b5c20_8100_4365_b3d6_a98996cb658b.slice/crio-895de872d53bfcf02c911c3ef3109451eae7fc8d59cf670cb73013c05bdc04ab WatchSource:0}: Error finding container 895de872d53bfcf02c911c3ef3109451eae7fc8d59cf670cb73013c05bdc04ab: Status 404 returned error can't find the container with id 895de872d53bfcf02c911c3ef3109451eae7fc8d59cf670cb73013c05bdc04ab Sep 18 17:51:02.205867 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.205802 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrn47"] Sep 18 17:51:02.209752 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:02.209730 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a79f05_3f65_487b_8eed_f9ffd65acf95.slice/crio-e5b470f451db9d1a67fd17872ef0a65ad7a7af641e72998375b2b7d271bd0454 WatchSource:0}: Error finding container e5b470f451db9d1a67fd17872ef0a65ad7a7af641e72998375b2b7d271bd0454: Status 404 returned error can't find the container with id e5b470f451db9d1a67fd17872ef0a65ad7a7af641e72998375b2b7d271bd0454 Sep 18 17:51:02.268514 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.268490 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrn47" event={"ID":"55a79f05-3f65-487b-8eed-f9ffd65acf95","Type":"ContainerStarted","Data":"e5b470f451db9d1a67fd17872ef0a65ad7a7af641e72998375b2b7d271bd0454"} Sep 18 17:51:02.269680 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.269660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9vfvg" event={"ID":"bf3b5c20-8100-4365-b3d6-a98996cb658b","Type":"ContainerStarted","Data":"895de872d53bfcf02c911c3ef3109451eae7fc8d59cf670cb73013c05bdc04ab"} Sep 18 17:51:02.270717 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.270699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58kvg" event={"ID":"5958b2d8-31bd-4b8d-8672-024520ce6ef6","Type":"ContainerStarted","Data":"53f59cc839152bb41c11bf8840716d6b5e4781a8110a9973d59daf510c8052d7"} Sep 18 17:51:02.335112 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.335086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:02.335241 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.335128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:51:02.337329 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.337306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9172cf61-f370-4720-9191-b5ea2f68d55e-metrics-tls\") pod \"dns-default-5ll45\" (UID: \"9172cf61-f370-4720-9191-b5ea2f68d55e\") " pod="openshift-dns/dns-default-5ll45" Sep 18 17:51:02.337413 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.337353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"image-registry-657d898764-rxf7r\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:02.601351 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.601321 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xssnq\"" Sep 18 17:51:02.608606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.608588 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:02.614308 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.614276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s9cmf\"" Sep 18 17:51:02.620965 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.620943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ll45" Sep 18 17:51:02.892373 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.891890 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:51:02.895851 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:02.895820 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29657b39_949d_4569_805a_6122d7c864f0.slice/crio-7c44cee3fd53b32d9fb69cc65f46f4a437e77c4fc3e62453482515effee11379 WatchSource:0}: Error finding container 7c44cee3fd53b32d9fb69cc65f46f4a437e77c4fc3e62453482515effee11379: Status 404 returned error can't find the container with id 7c44cee3fd53b32d9fb69cc65f46f4a437e77c4fc3e62453482515effee11379 Sep 18 17:51:02.916561 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:02.916533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ll45"] Sep 18 17:51:02.923742 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:02.923708 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9172cf61_f370_4720_9191_b5ea2f68d55e.slice/crio-23ecaa33862ed378ba63e14ef7d952a49e13d8b60c8247163e71715f0985d809 WatchSource:0}: Error finding container 23ecaa33862ed378ba63e14ef7d952a49e13d8b60c8247163e71715f0985d809: Status 404 returned error can't find the container with id 23ecaa33862ed378ba63e14ef7d952a49e13d8b60c8247163e71715f0985d809 Sep 18 17:51:03.278602 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:03.278513 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ll45" event={"ID":"9172cf61-f370-4720-9191-b5ea2f68d55e","Type":"ContainerStarted","Data":"23ecaa33862ed378ba63e14ef7d952a49e13d8b60c8247163e71715f0985d809"} Sep 18 17:51:03.280903 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:03.280872 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657d898764-rxf7r" event={"ID":"29657b39-949d-4569-805a-6122d7c864f0","Type":"ContainerStarted","Data":"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb"} Sep 18 17:51:03.281038 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:03.280929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657d898764-rxf7r" event={"ID":"29657b39-949d-4569-805a-6122d7c864f0","Type":"ContainerStarted","Data":"7c44cee3fd53b32d9fb69cc65f46f4a437e77c4fc3e62453482515effee11379"} Sep 18 17:51:03.281733 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:03.281709 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:03.315048 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:03.314264 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657d898764-rxf7r" podStartSLOduration=43.314239091 podStartE2EDuration="43.314239091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:03.313262753 +0000 UTC m=+68.405917860" watchObservedRunningTime="2026-09-18 17:51:03.314239091 +0000 UTC m=+68.406894182" Sep 18 17:51:04.288473 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.288435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrn47" event={"ID":"55a79f05-3f65-487b-8eed-f9ffd65acf95","Type":"ContainerStarted","Data":"1dc56e84417e91a30900ade0fe9dd254354542f75aeafe4e30332d31c750a407"} Sep 18 17:51:04.288954 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.288502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrn47" event={"ID":"55a79f05-3f65-487b-8eed-f9ffd65acf95","Type":"ContainerStarted","Data":"5b8351f6692df229c5d3adcea7954636c5ed3c4a405b4d51a7f062ff8d36df5b"} Sep 18 17:51:04.335149 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.335063 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vrn47" podStartSLOduration=66.098183438 podStartE2EDuration="1m7.33504843s" totalImagesPullingTime="1.236864992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:04.333615105 +0000 UTC m=+69.426270239" watchObservedRunningTime="2026-09-18 17:51:04.33504843 +0000 UTC m=+69.427703520" Sep 18 17:51:04.656318 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.656236 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc"] Sep 18 17:51:04.659274 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.659002 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-g7blk"] Sep 18 17:51:04.659274 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.659263 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.661241 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.661006 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb"] Sep 18 17:51:04.662073 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.661447 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.663386 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.662830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.663682 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.663594 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 18 17:51:04.663682 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.663634 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-68psr\"" Sep 18 17:51:04.663819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.663639 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 17:51:04.663971 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.663945 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 17:51:04.669288 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.669226 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.669553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670102 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670347 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670489 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670541 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2kpnm\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670585 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 18 17:51:04.671146 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.670761 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 18 17:51:04.677481 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.677441 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc"] Sep 18 17:51:04.700588 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.700173 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb"] Sep 18 17:51:04.700588 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.700251 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-g7blk"] Sep 18 17:51:04.705252 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.705226 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:51:04.750352 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxs8\" (UniqueName: \"kubernetes.io/projected/a06f5108-070f-4c54-9965-c8698f8c6241-kube-api-access-2lxs8\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxz7v\" (UniqueName: \"kubernetes.io/projected/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-kube-api-access-cxz7v\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750449 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06f5108-070f-4c54-9965-c8698f8c6241-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.750537 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c151da0-f847-400f-89c7-7b5fdaca26de-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.750845 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.750845 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750579 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c151da0-f847-400f-89c7-7b5fdaca26de-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.750845 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.750596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851074 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c151da0-f847-400f-89c7-7b5fdaca26de-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c151da0-f847-400f-89c7-7b5fdaca26de-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851238 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851503 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxs8\" (UniqueName: \"kubernetes.io/projected/a06f5108-070f-4c54-9965-c8698f8c6241-kube-api-access-2lxs8\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.851503 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxz7v\" (UniqueName: \"kubernetes.io/projected/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-kube-api-access-cxz7v\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.851503 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.851309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06f5108-070f-4c54-9965-c8698f8c6241-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.852151 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.852096 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06f5108-070f-4c54-9965-c8698f8c6241-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.852151 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.852147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c151da0-f847-400f-89c7-7b5fdaca26de-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.854838 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.854790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.854956 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.854906 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-ca\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.855520 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.855465 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c151da0-f847-400f-89c7-7b5fdaca26de-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-g7blk\" (UID: \"0c151da0-f847-400f-89c7-7b5fdaca26de\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.856461 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.856416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.856742 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.856720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.857006 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.856981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a06f5108-070f-4c54-9965-c8698f8c6241-hub\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.870033 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.869974 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxs8\" (UniqueName: \"kubernetes.io/projected/a06f5108-070f-4c54-9965-c8698f8c6241-kube-api-access-2lxs8\") pod \"cluster-proxy-proxy-agent-6ccf989745-ckfmb\" (UID: \"a06f5108-070f-4c54-9965-c8698f8c6241\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:04.870650 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.870627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxz7v\" (UniqueName: \"kubernetes.io/projected/c70e42bf-b28b-4656-b250-5bc0b7b52ba4-kube-api-access-cxz7v\") pod \"managed-serviceaccount-addon-agent-588c7d59f8-wszvc\" (UID: \"c70e42bf-b28b-4656-b250-5bc0b7b52ba4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.980500 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.980427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" Sep 18 17:51:04.990657 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.990625 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" Sep 18 17:51:04.998582 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:04.998557 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" Sep 18 17:51:07.685410 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:07.685368 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc"] Sep 18 17:51:07.687986 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:07.687961 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70e42bf_b28b_4656_b250_5bc0b7b52ba4.slice/crio-13e77b4286e2174baf7f57c49cc0fca38a1be43388afd39c9ede0e4dae348db6 WatchSource:0}: Error finding container 13e77b4286e2174baf7f57c49cc0fca38a1be43388afd39c9ede0e4dae348db6: Status 404 returned error can't find the container with id 13e77b4286e2174baf7f57c49cc0fca38a1be43388afd39c9ede0e4dae348db6 Sep 18 17:51:07.732686 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:07.732663 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb"] Sep 18 17:51:07.733445 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:07.733424 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-g7blk"] Sep 18 17:51:07.737265 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:07.737242 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda06f5108_070f_4c54_9965_c8698f8c6241.slice/crio-5b4b00d76b1395ab6eca137d238edce7a987b139ca5ccf3dd6c5f4d19ba6b5cb WatchSource:0}: Error finding container 5b4b00d76b1395ab6eca137d238edce7a987b139ca5ccf3dd6c5f4d19ba6b5cb: Status 404 returned error can't find the container with id 5b4b00d76b1395ab6eca137d238edce7a987b139ca5ccf3dd6c5f4d19ba6b5cb Sep 18 17:51:07.737916 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:07.737862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c151da0_f847_400f_89c7_7b5fdaca26de.slice/crio-b563c942acf1e5a19445d3a9d325bfb83d59fd77e16a1e9186b8b7c85ec3596b WatchSource:0}: Error finding container b563c942acf1e5a19445d3a9d325bfb83d59fd77e16a1e9186b8b7c85ec3596b: Status 404 returned error can't find the container with id b563c942acf1e5a19445d3a9d325bfb83d59fd77e16a1e9186b8b7c85ec3596b Sep 18 17:51:08.305085 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.305048 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" event={"ID":"c70e42bf-b28b-4656-b250-5bc0b7b52ba4","Type":"ContainerStarted","Data":"13e77b4286e2174baf7f57c49cc0fca38a1be43388afd39c9ede0e4dae348db6"} Sep 18 17:51:08.306507 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.306478 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9vfvg" event={"ID":"bf3b5c20-8100-4365-b3d6-a98996cb658b","Type":"ContainerStarted","Data":"e2b68b8351228304d1635ccd18a5ff041337744f65beb17d330a08d60c865323"} Sep 18 17:51:08.306683 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.306544 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:51:08.308144 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.308112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58kvg" event={"ID":"5958b2d8-31bd-4b8d-8672-024520ce6ef6","Type":"ContainerStarted","Data":"666b2adc4af469433a786f8151b6ff35e91aca95568ff11c03e5efd5e3baa77a"} Sep 18 17:51:08.309165 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.309143 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" event={"ID":"a06f5108-070f-4c54-9965-c8698f8c6241","Type":"ContainerStarted","Data":"5b4b00d76b1395ab6eca137d238edce7a987b139ca5ccf3dd6c5f4d19ba6b5cb"} Sep 18 17:51:08.310750 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.310728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ll45" event={"ID":"9172cf61-f370-4720-9191-b5ea2f68d55e","Type":"ContainerStarted","Data":"88714dd514d9ad416160f850c1874e263348ca80bf9f47bf54acc33b6cfe8210"} Sep 18 17:51:08.310833 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.310771 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ll45" event={"ID":"9172cf61-f370-4720-9191-b5ea2f68d55e","Type":"ContainerStarted","Data":"91500865896d06d116c0f0710d91ac844d26d6323eaac4aa8ced816da746017e"} Sep 18 17:51:08.310833 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.310796 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5ll45" Sep 18 17:51:08.311788 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.311770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" event={"ID":"0c151da0-f847-400f-89c7-7b5fdaca26de","Type":"ContainerStarted","Data":"b563c942acf1e5a19445d3a9d325bfb83d59fd77e16a1e9186b8b7c85ec3596b"} Sep 18 17:51:08.353303 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.353235 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9vfvg" podStartSLOduration=66.159200889 podStartE2EDuration="1m11.353219791s" totalImagesPullingTime="5.194018902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:08.352569414 +0000 UTC m=+73.445224519" watchObservedRunningTime="2026-09-18 17:51:08.353219791 +0000 UTC m=+73.445874878" Sep 18 17:51:08.414842 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.414793 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-58kvg" podStartSLOduration=66.181051763 podStartE2EDuration="1m11.414746558s" totalImagesPullingTime="5.233694795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:08.409583345 +0000 UTC m=+73.502238453" watchObservedRunningTime="2026-09-18 17:51:08.414746558 +0000 UTC m=+73.507401644" Sep 18 17:51:08.468158 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:08.467536 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5ll45" podStartSLOduration=34.02340738 podStartE2EDuration="38.467512158s" totalImagesPullingTime="4.444104778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:08.465893417 +0000 UTC m=+73.558548525" watchObservedRunningTime="2026-09-18 17:51:08.467512158 +0000 UTC m=+73.560167246" Sep 18 17:51:11.325804 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:11.325767 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" event={"ID":"0c151da0-f847-400f-89c7-7b5fdaca26de","Type":"ContainerStarted","Data":"4f0edddde96bfcdc6b3fcdc20c967a3ee1915d01b733b1d6c5ca9dfe6949c509"} Sep 18 17:51:11.327341 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:11.327316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" event={"ID":"c70e42bf-b28b-4656-b250-5bc0b7b52ba4","Type":"ContainerStarted","Data":"2add75d8a04025d313ac225a8fba00652b8b9e4234fab2bc6d980626157531b6"} Sep 18 17:51:11.359535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:11.359458 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-g7blk" podStartSLOduration=4.9575858010000005 podStartE2EDuration="7.359438724s" totalImagesPullingTime="2.401852923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:11.358412341 +0000 UTC m=+76.451067449" watchObservedRunningTime="2026-09-18 17:51:11.359438724 +0000 UTC m=+76.452093812" Sep 18 17:51:11.385386 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:11.385325 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588c7d59f8-wszvc" podStartSLOduration=4.52442213 podStartE2EDuration="7.385303298s" totalImagesPullingTime="2.860881168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:11.384710349 +0000 UTC m=+76.477365437" watchObservedRunningTime="2026-09-18 17:51:11.385303298 +0000 UTC m=+76.477958386" Sep 18 17:51:13.336780 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:13.336694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" event={"ID":"a06f5108-070f-4c54-9965-c8698f8c6241","Type":"ContainerStarted","Data":"e7d7186e6af4227e0b436a9fe4a321980f020c29e4e67fe3a15340d07214c9a9"} Sep 18 17:51:13.336780 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:13.336739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" event={"ID":"a06f5108-070f-4c54-9965-c8698f8c6241","Type":"ContainerStarted","Data":"5b1bd0f6f9852bd2d560be1574b4146aa5d8ae5aab2ee571042fd41c6dd18000"} Sep 18 17:51:13.336780 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:13.336752 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" event={"ID":"a06f5108-070f-4c54-9965-c8698f8c6241","Type":"ContainerStarted","Data":"02a5f2c0bc838dfbe5e54a9cd7ca6b0c5dc0409b60830b8326b9e0734b0736f4"} Sep 18 17:51:18.318980 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:18.318951 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5ll45" Sep 18 17:51:18.360047 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:18.360001 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ccf989745-ckfmb" podStartSLOduration=9.14040279 podStartE2EDuration="14.359985549s" totalImagesPullingTime="5.219582759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:13.365561913 +0000 UTC m=+78.458217000" watchObservedRunningTime="2026-09-18 17:51:18.359985549 +0000 UTC m=+83.452640669" Sep 18 17:51:26.303078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:26.303045 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:27.574900 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.574867 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x9bq6"] Sep 18 17:51:27.579311 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.579290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.582685 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.582657 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 17:51:27.582962 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.582938 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 17:51:27.583519 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.583304 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 17:51:27.583690 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.583616 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 17:51:27.583690 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.583678 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gg4sf\"" Sep 18 17:51:27.585940 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.585837 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 17:51:27.586846 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.586829 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 17:51:27.593874 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.593854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.593961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.593889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-wtmp\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.593961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.593923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594042 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.593993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594042 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.594023 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-sys\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594107 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.594044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-textfile\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594107 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.594060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6pbs\" (UniqueName: \"kubernetes.io/projected/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-kube-api-access-d6pbs\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.594099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-root\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.594226 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.594178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-metrics-client-ca\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695160 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-wtmp\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-sys\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695308 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-textfile\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6pbs\" (UniqueName: \"kubernetes.io/projected/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-kube-api-access-d6pbs\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695344 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-wtmp\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-root\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695391 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-sys\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:51:27.695407 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695404 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-metrics-client-ca\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:51:27.695477 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls podName:728290a5-8d19-4b0f-81ba-11f2a6fba8ca nodeName:}" failed. No retries permitted until 2026-09-18 17:51:28.195455838 +0000 UTC m=+93.288110926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls") pod "node-exporter-x9bq6" (UID: "728290a5-8d19-4b0f-81ba-11f2a6fba8ca") : secret "node-exporter-tls" not found Sep 18 17:51:27.695567 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-root\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695885 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-textfile\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695885 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695736 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.695885 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.695807 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-metrics-client-ca\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.697577 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.697560 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:27.715316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:27.715290 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6pbs\" (UniqueName: \"kubernetes.io/projected/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-kube-api-access-d6pbs\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:28.199458 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:28.199426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:28.201639 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:28.201621 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/728290a5-8d19-4b0f-81ba-11f2a6fba8ca-node-exporter-tls\") pod \"node-exporter-x9bq6\" (UID: \"728290a5-8d19-4b0f-81ba-11f2a6fba8ca\") " pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:28.489383 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:28.489300 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9bq6" Sep 18 17:51:28.499861 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:51:28.499831 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728290a5_8d19_4b0f_81ba_11f2a6fba8ca.slice/crio-633b416999ca1787fb0677e82a8d0713361bac0f235a2c9bb41b4b840dc746b2 WatchSource:0}: Error finding container 633b416999ca1787fb0677e82a8d0713361bac0f235a2c9bb41b4b840dc746b2: Status 404 returned error can't find the container with id 633b416999ca1787fb0677e82a8d0713361bac0f235a2c9bb41b4b840dc746b2 Sep 18 17:51:29.392466 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:29.392401 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9bq6" event={"ID":"728290a5-8d19-4b0f-81ba-11f2a6fba8ca","Type":"ContainerStarted","Data":"633b416999ca1787fb0677e82a8d0713361bac0f235a2c9bb41b4b840dc746b2"} Sep 18 17:51:30.398687 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:30.398655 2570 generic.go:356] "Generic (PLEG): container finished" podID="728290a5-8d19-4b0f-81ba-11f2a6fba8ca" containerID="c06d09a0b9297c09b957b0ce8236eca1b5c2570916b7a69c16b8f73d044064c9" exitCode=0 Sep 18 17:51:30.399084 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:30.398696 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9bq6" event={"ID":"728290a5-8d19-4b0f-81ba-11f2a6fba8ca","Type":"ContainerDied","Data":"c06d09a0b9297c09b957b0ce8236eca1b5c2570916b7a69c16b8f73d044064c9"} Sep 18 17:51:31.316874 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.316804 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-657d898764-rxf7r" podUID="29657b39-949d-4569-805a-6122d7c864f0" containerName="registry" containerID="cri-o://f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb" gracePeriod=30 Sep 18 17:51:31.404497 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.404466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9bq6" event={"ID":"728290a5-8d19-4b0f-81ba-11f2a6fba8ca","Type":"ContainerStarted","Data":"075720d9281320a294b3e6b8c001d7c1bde6d1193c5c4c9cb39584948e8a741c"} Sep 18 17:51:31.404909 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.404540 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9bq6" event={"ID":"728290a5-8d19-4b0f-81ba-11f2a6fba8ca","Type":"ContainerStarted","Data":"50613dd38c0870e8d2c9cb86b62163ea20dba23a3a628ae79e30da6bc3a6d5c0"} Sep 18 17:51:31.553825 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.553799 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:31.577755 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.577669 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x9bq6" podStartSLOduration=3.504887058 podStartE2EDuration="4.577650054s" totalImagesPullingTime="1.0129821s" totalInitContainerRuntime="59.780896ms" isStatefulPod=true podCreationTimestamp="2026-09-18 17:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:31.433063286 +0000 UTC m=+96.525718403" watchObservedRunningTime="2026-09-18 17:51:31.577650054 +0000 UTC m=+96.670305142" Sep 18 17:51:31.627702 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627675 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.627819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627707 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.627819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627735 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.627819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627776 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.627819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627806 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4prm\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.628017 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627823 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.628017 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627855 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.628017 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.627886 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") pod \"29657b39-949d-4569-805a-6122d7c864f0\" (UID: \"29657b39-949d-4569-805a-6122d7c864f0\") " Sep 18 17:51:31.628263 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.628231 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:51:31.628363 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.628258 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:51:31.630568 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.630538 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:51:31.630701 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.630544 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:51:31.630701 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.630643 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:51:31.630819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.630726 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:51:31.630819 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.630802 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm" (OuterVolumeSpecName: "kube-api-access-v4prm") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "kube-api-access-v4prm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:51:31.635984 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.635963 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "29657b39-949d-4569-805a-6122d7c864f0" (UID: "29657b39-949d-4569-805a-6122d7c864f0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:51:31.729229 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729177 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-trusted-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729229 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729230 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29657b39-949d-4569-805a-6122d7c864f0-registry-certificates\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729242 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-image-registry-private-configuration\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729252 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29657b39-949d-4569-805a-6122d7c864f0-installation-pull-secrets\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729261 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4prm\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-kube-api-access-v4prm\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729269 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-bound-sa-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729276 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29657b39-949d-4569-805a-6122d7c864f0-ca-trust-extracted\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:31.729375 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:31.729284 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29657b39-949d-4569-805a-6122d7c864f0-registry-tls\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:51:32.409372 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.409328 2570 generic.go:356] "Generic (PLEG): container finished" podID="29657b39-949d-4569-805a-6122d7c864f0" containerID="f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb" exitCode=0 Sep 18 17:51:32.409764 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.409382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657d898764-rxf7r" Sep 18 17:51:32.409764 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.409410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657d898764-rxf7r" event={"ID":"29657b39-949d-4569-805a-6122d7c864f0","Type":"ContainerDied","Data":"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb"} Sep 18 17:51:32.409764 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.409464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657d898764-rxf7r" event={"ID":"29657b39-949d-4569-805a-6122d7c864f0","Type":"ContainerDied","Data":"7c44cee3fd53b32d9fb69cc65f46f4a437e77c4fc3e62453482515effee11379"} Sep 18 17:51:32.409764 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.409484 2570 scope.go:117] "RemoveContainer" containerID="f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb" Sep 18 17:51:32.417664 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.417643 2570 scope.go:117] "RemoveContainer" containerID="f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb" Sep 18 17:51:32.417932 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:51:32.417913 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb\": container with ID starting with f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb not found: ID does not exist" containerID="f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb" Sep 18 17:51:32.417990 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.417940 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb"} err="failed to get container status \"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb\": rpc error: code = NotFound desc = could not find container \"f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb\": container with ID starting with f5ef44249229fc230dd13ffa68605cd78a05b8da00884c7dc3e0fdb318ee9ddb not found: ID does not exist" Sep 18 17:51:32.434260 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.434239 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:51:32.444218 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.444175 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-657d898764-rxf7r"] Sep 18 17:51:32.941125 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:32.941096 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29657b39-949d-4569-805a-6122d7c864f0" path="/var/lib/kubelet/pods/29657b39-949d-4569-805a-6122d7c864f0/volumes" Sep 18 17:51:39.319860 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:51:39.319828 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9vfvg" Sep 18 17:54:56.863132 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:54:56.863103 2570 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 17:55:10.258554 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.258514 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6"] Sep 18 17:55:10.259926 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.258746 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29657b39-949d-4569-805a-6122d7c864f0" containerName="registry" Sep 18 17:55:10.259926 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.258760 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29657b39-949d-4569-805a-6122d7c864f0" containerName="registry" Sep 18 17:55:10.259926 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.258806 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29657b39-949d-4569-805a-6122d7c864f0" containerName="registry" Sep 18 17:55:10.260707 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.260690 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.271344 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.271325 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:55:10.271451 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.271358 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:55:10.272031 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.272017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q2zwl\"" Sep 18 17:55:10.285495 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.285462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6"] Sep 18 17:55:10.346369 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.346341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thsl9\" (UniqueName: \"kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.346369 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.346373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.346558 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.346402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.447757 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.447729 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thsl9\" (UniqueName: \"kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.447863 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.447762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.447863 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.447789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.448121 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.448101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.448205 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.448126 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.459744 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.459714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thsl9\" (UniqueName: \"kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.568851 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.568763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:10.854390 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.854308 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6"] Sep 18 17:55:10.858599 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:55:10.858571 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7adfe7_ef98_44e7_8ec1_1670bf551e07.slice/crio-b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538 WatchSource:0}: Error finding container b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538: Status 404 returned error can't find the container with id b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538 Sep 18 17:55:10.860439 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:10.860420 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:55:11.097810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:11.097778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerStarted","Data":"b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538"} Sep 18 17:55:18.123935 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:18.123898 2570 generic.go:356] "Generic (PLEG): container finished" podID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerID="b7dc7ccd94294a49b2f647c1e8b6ec2c30d8ca997757caf3e216875da657873e" exitCode=0 Sep 18 17:55:18.124333 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:18.123993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerDied","Data":"b7dc7ccd94294a49b2f647c1e8b6ec2c30d8ca997757caf3e216875da657873e"} Sep 18 17:55:20.132309 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:20.132279 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerStarted","Data":"2349ef05b413f665f2544feaf0aa0748ebc565d5795607585ec6468362e83d9d"} Sep 18 17:55:21.137344 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:21.137310 2570 generic.go:356] "Generic (PLEG): container finished" podID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerID="2349ef05b413f665f2544feaf0aa0748ebc565d5795607585ec6468362e83d9d" exitCode=0 Sep 18 17:55:21.137726 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:21.137349 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerDied","Data":"2349ef05b413f665f2544feaf0aa0748ebc565d5795607585ec6468362e83d9d"} Sep 18 17:55:28.162458 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:28.162425 2570 generic.go:356] "Generic (PLEG): container finished" podID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerID="bda71bef214d80361104c1cbbbc3080333c9d57f76ac734f3de32027acbb8c3d" exitCode=0 Sep 18 17:55:28.162769 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:28.162509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerDied","Data":"bda71bef214d80361104c1cbbbc3080333c9d57f76ac734f3de32027acbb8c3d"} Sep 18 17:55:29.286967 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.286941 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:29.379216 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.379158 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thsl9\" (UniqueName: \"kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9\") pod \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " Sep 18 17:55:29.379386 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.379231 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util\") pod \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " Sep 18 17:55:29.379386 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.379334 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle\") pod \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\" (UID: \"0e7adfe7-ef98-44e7-8ec1-1670bf551e07\") " Sep 18 17:55:29.379894 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.379871 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle" (OuterVolumeSpecName: "bundle") pod "0e7adfe7-ef98-44e7-8ec1-1670bf551e07" (UID: "0e7adfe7-ef98-44e7-8ec1-1670bf551e07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:55:29.381221 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.381180 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9" (OuterVolumeSpecName: "kube-api-access-thsl9") pod "0e7adfe7-ef98-44e7-8ec1-1670bf551e07" (UID: "0e7adfe7-ef98-44e7-8ec1-1670bf551e07"). InnerVolumeSpecName "kube-api-access-thsl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:55:29.383533 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.383510 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util" (OuterVolumeSpecName: "util") pod "0e7adfe7-ef98-44e7-8ec1-1670bf551e07" (UID: "0e7adfe7-ef98-44e7-8ec1-1670bf551e07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:55:29.480714 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.480640 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:29.480714 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.480667 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:29.480714 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:29.480679 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thsl9\" (UniqueName: \"kubernetes.io/projected/0e7adfe7-ef98-44e7-8ec1-1670bf551e07-kube-api-access-thsl9\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:30.171138 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:30.171110 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" Sep 18 17:55:30.171138 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:30.171116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mwlc6" event={"ID":"0e7adfe7-ef98-44e7-8ec1-1670bf551e07","Type":"ContainerDied","Data":"b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538"} Sep 18 17:55:30.171369 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:30.171162 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9173297f2d66884c93e5a592602e29d27720480e7525351baae186e3e641538" Sep 18 17:55:34.418134 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418092 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c"] Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418419 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="pull" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418437 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="pull" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418454 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="extract" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418463 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="extract" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418481 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="util" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418490 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="util" Sep 18 17:55:34.418606 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.418544 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7adfe7-ef98-44e7-8ec1-1670bf551e07" containerName="extract" Sep 18 17:55:34.423563 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.422241 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.426918 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.426896 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:55:34.427263 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.427244 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qzzwr\"" Sep 18 17:55:34.427418 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.427405 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 18 17:55:34.449918 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.449895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c"] Sep 18 17:55:34.512979 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.512950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a86d428d-78b8-477f-a513-de3495c938a5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.513082 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.512984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnkg\" (UniqueName: \"kubernetes.io/projected/a86d428d-78b8-477f-a513-de3495c938a5-kube-api-access-prnkg\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.614121 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.614087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a86d428d-78b8-477f-a513-de3495c938a5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.614288 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.614125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnkg\" (UniqueName: \"kubernetes.io/projected/a86d428d-78b8-477f-a513-de3495c938a5-kube-api-access-prnkg\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.614565 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.614540 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a86d428d-78b8-477f-a513-de3495c938a5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.623645 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.623622 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnkg\" (UniqueName: \"kubernetes.io/projected/a86d428d-78b8-477f-a513-de3495c938a5-kube-api-access-prnkg\") pod \"cert-manager-operator-controller-manager-55c444bf86-hmk9c\" (UID: \"a86d428d-78b8-477f-a513-de3495c938a5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:34.731372 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:34.731305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" Sep 18 17:55:35.016783 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:35.016753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c"] Sep 18 17:55:35.018604 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:55:35.018581 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86d428d_78b8_477f_a513_de3495c938a5.slice/crio-7d7ffb469050b781ca77d674662319dfbdb80703deeecd55a701ef5e49d618e1 WatchSource:0}: Error finding container 7d7ffb469050b781ca77d674662319dfbdb80703deeecd55a701ef5e49d618e1: Status 404 returned error can't find the container with id 7d7ffb469050b781ca77d674662319dfbdb80703deeecd55a701ef5e49d618e1 Sep 18 17:55:35.187806 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:35.187769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" event={"ID":"a86d428d-78b8-477f-a513-de3495c938a5","Type":"ContainerStarted","Data":"7d7ffb469050b781ca77d674662319dfbdb80703deeecd55a701ef5e49d618e1"} Sep 18 17:55:37.195354 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:37.195321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" event={"ID":"a86d428d-78b8-477f-a513-de3495c938a5","Type":"ContainerStarted","Data":"5d652f224e49750708fac693ee90212acc6d46feeaa5bd09cf0311b6436b8303"} Sep 18 17:55:37.225653 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:37.225604 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hmk9c" podStartSLOduration=1.490124147 podStartE2EDuration="3.225587606s" totalImagesPullingTime="1.735463459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:55:37.223592262 +0000 UTC m=+342.316247367" watchObservedRunningTime="2026-09-18 17:55:37.225587606 +0000 UTC m=+342.318242693" Sep 18 17:55:38.136620 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.136580 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx"] Sep 18 17:55:38.139945 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.139921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.144728 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.144708 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:55:38.144982 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.144963 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q2zwl\"" Sep 18 17:55:38.145558 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.145539 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:55:38.185707 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.185684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx"] Sep 18 17:55:38.237329 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.237300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsld4\" (UniqueName: \"kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.237652 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.237347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.237652 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.237378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.338368 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.338339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.338494 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.338382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.338494 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.338403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsld4\" (UniqueName: \"kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.338774 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.338755 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.338810 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.338758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.366743 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.366709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsld4\" (UniqueName: \"kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.448158 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.448130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:38.779264 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:38.777889 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx"] Sep 18 17:55:38.783416 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:55:38.783385 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b2af6f_79df_4362_8030_136e7a44b38e.slice/crio-8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c WatchSource:0}: Error finding container 8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c: Status 404 returned error can't find the container with id 8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c Sep 18 17:55:39.172371 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.172339 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8297n"] Sep 18 17:55:39.175168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.175153 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.178170 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.178149 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r2d4r\"" Sep 18 17:55:39.178844 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.178824 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 18 17:55:39.179492 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.179472 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 18 17:55:39.190804 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.190784 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8297n"] Sep 18 17:55:39.202847 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.202824 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerID="9629929f841795ce2e06570a3df447109be749b712d676ac80a89f06b12338e5" exitCode=0 Sep 18 17:55:39.202942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.202886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerDied","Data":"9629929f841795ce2e06570a3df447109be749b712d676ac80a89f06b12338e5"} Sep 18 17:55:39.202942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.202933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerStarted","Data":"8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c"} Sep 18 17:55:39.244826 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.244773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.244826 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.244819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sgs4\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-kube-api-access-6sgs4\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.345341 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.345316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgs4\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-kube-api-access-6sgs4\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.345450 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.345371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.376301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.376278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.376442 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.376425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgs4\" (UniqueName: \"kubernetes.io/projected/f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3-kube-api-access-6sgs4\") pod \"cert-manager-webhook-6486ffd997-8297n\" (UID: \"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.483508 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.483437 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:39.765280 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:39.765210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8297n"] Sep 18 17:55:39.768336 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:55:39.768309 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e40e0f_87c4_4e9f_ab16_bd7e570bb8a3.slice/crio-19294fb80d3858c969f37fd3d7cfe20a0950868d0ece58f00d46338b96a9a6c3 WatchSource:0}: Error finding container 19294fb80d3858c969f37fd3d7cfe20a0950868d0ece58f00d46338b96a9a6c3: Status 404 returned error can't find the container with id 19294fb80d3858c969f37fd3d7cfe20a0950868d0ece58f00d46338b96a9a6c3 Sep 18 17:55:40.208562 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:40.208524 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" event={"ID":"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3","Type":"ContainerStarted","Data":"19294fb80d3858c969f37fd3d7cfe20a0950868d0ece58f00d46338b96a9a6c3"} Sep 18 17:55:42.220165 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:42.220131 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerID="0d9e272ecfb27c5630b916e15cf6206aa3834cf9f5b6f9698b517fbbf46810ab" exitCode=0 Sep 18 17:55:42.220547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:42.220213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerDied","Data":"0d9e272ecfb27c5630b916e15cf6206aa3834cf9f5b6f9698b517fbbf46810ab"} Sep 18 17:55:43.227231 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:43.227163 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerStarted","Data":"96faf2621c2dc214e00c4c25bdeaadd39431a5e08cb50222754228f509bb9515"} Sep 18 17:55:43.229120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:43.229086 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" event={"ID":"f2e40e0f-87c4-4e9f-ab16-bd7e570bb8a3","Type":"ContainerStarted","Data":"3dfec2b1d695ea2b7fda41672e46f5406b7612df5d4220882e4c0e7ce38349b0"} Sep 18 17:55:43.229293 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:43.229163 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:43.259578 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:43.259527 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" podStartSLOduration=3.2419478330000002 podStartE2EDuration="5.259511596s" totalImagesPullingTime="2.017563763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:55:43.257584724 +0000 UTC m=+348.350239831" watchObservedRunningTime="2026-09-18 17:55:43.259511596 +0000 UTC m=+348.352166682" Sep 18 17:55:43.284226 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:43.284156 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" podStartSLOduration=0.895572086 podStartE2EDuration="4.284140978s" totalImagesPullingTime="3.388568892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:55:43.283553566 +0000 UTC m=+348.376208653" watchObservedRunningTime="2026-09-18 17:55:43.284140978 +0000 UTC m=+348.376796065" Sep 18 17:55:44.234120 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:44.234081 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerID="96faf2621c2dc214e00c4c25bdeaadd39431a5e08cb50222754228f509bb9515" exitCode=0 Sep 18 17:55:44.234497 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:44.234167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerDied","Data":"96faf2621c2dc214e00c4c25bdeaadd39431a5e08cb50222754228f509bb9515"} Sep 18 17:55:45.364641 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.364617 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:45.387745 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.387716 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util\") pod \"a0b2af6f-79df-4362-8030-136e7a44b38e\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " Sep 18 17:55:45.387906 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.387774 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle\") pod \"a0b2af6f-79df-4362-8030-136e7a44b38e\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " Sep 18 17:55:45.387906 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.387809 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsld4\" (UniqueName: \"kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4\") pod \"a0b2af6f-79df-4362-8030-136e7a44b38e\" (UID: \"a0b2af6f-79df-4362-8030-136e7a44b38e\") " Sep 18 17:55:45.388234 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.388171 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle" (OuterVolumeSpecName: "bundle") pod "a0b2af6f-79df-4362-8030-136e7a44b38e" (UID: "a0b2af6f-79df-4362-8030-136e7a44b38e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:55:45.390487 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.390455 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4" (OuterVolumeSpecName: "kube-api-access-hsld4") pod "a0b2af6f-79df-4362-8030-136e7a44b38e" (UID: "a0b2af6f-79df-4362-8030-136e7a44b38e"). InnerVolumeSpecName "kube-api-access-hsld4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:55:45.394429 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.394402 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util" (OuterVolumeSpecName: "util") pod "a0b2af6f-79df-4362-8030-136e7a44b38e" (UID: "a0b2af6f-79df-4362-8030-136e7a44b38e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:55:45.488798 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.488722 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsld4\" (UniqueName: \"kubernetes.io/projected/a0b2af6f-79df-4362-8030-136e7a44b38e-kube-api-access-hsld4\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:45.488798 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.488750 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:45.488798 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:45.488759 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0b2af6f-79df-4362-8030-136e7a44b38e-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:55:46.242195 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:46.242155 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" Sep 18 17:55:46.242372 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:46.242148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7mtx" event={"ID":"a0b2af6f-79df-4362-8030-136e7a44b38e","Type":"ContainerDied","Data":"8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c"} Sep 18 17:55:46.242372 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:46.242283 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1a3b3e61aa89bd0bf19822af6ed0d554768ac024e586b00fe9f2288e69010c" Sep 18 17:55:49.236344 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:49.236316 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8297n" Sep 18 17:55:58.029855 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.029820 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm"] Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030073 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="util" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030084 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="util" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030092 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="pull" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030098 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="pull" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030105 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="extract" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030111 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="extract" Sep 18 17:55:58.030301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.030146 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0b2af6f-79df-4362-8030-136e7a44b38e" containerName="extract" Sep 18 17:55:58.034114 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.034097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.036850 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.036826 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:55:58.036969 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.036925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q2zwl\"" Sep 18 17:55:58.036969 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.036955 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:55:58.043276 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.043255 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm"] Sep 18 17:55:58.072800 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.072774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6mj\" (UniqueName: \"kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.072903 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.072806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.072903 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.072854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.174076 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.174032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.174076 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.174085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6mj\" (UniqueName: \"kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.174296 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.174108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.174485 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.174469 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.174522 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.174489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.184677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.184651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6mj\" (UniqueName: \"kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.343811 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.343723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:55:58.622593 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:58.622506 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm"] Sep 18 17:55:58.626568 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:55:58.626541 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452c133b_e9db_4a10_86dd_eabb11c3b7a6.slice/crio-2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e WatchSource:0}: Error finding container 2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e: Status 404 returned error can't find the container with id 2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e Sep 18 17:55:59.288684 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:59.288651 2570 generic.go:356] "Generic (PLEG): container finished" podID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerID="8d7d1b762cd866a2deb5b976160e3eae7113eabd58cbc855bbed2e663e31f78f" exitCode=0 Sep 18 17:55:59.289012 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:59.288721 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" event={"ID":"452c133b-e9db-4a10-86dd-eabb11c3b7a6","Type":"ContainerDied","Data":"8d7d1b762cd866a2deb5b976160e3eae7113eabd58cbc855bbed2e663e31f78f"} Sep 18 17:55:59.289012 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:55:59.288769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" event={"ID":"452c133b-e9db-4a10-86dd-eabb11c3b7a6","Type":"ContainerStarted","Data":"2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e"} Sep 18 17:56:00.294594 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:00.294559 2570 generic.go:356] "Generic (PLEG): container finished" podID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerID="b25ccfeda0bd496bbdb42da0eabc0db370aec2c5dac8e5b8bc553ce6e34fb570" exitCode=0 Sep 18 17:56:00.295010 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:00.294654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" event={"ID":"452c133b-e9db-4a10-86dd-eabb11c3b7a6","Type":"ContainerDied","Data":"b25ccfeda0bd496bbdb42da0eabc0db370aec2c5dac8e5b8bc553ce6e34fb570"} Sep 18 17:56:01.300264 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:01.300229 2570 generic.go:356] "Generic (PLEG): container finished" podID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerID="4b303367bbfa02ffe0ccc30f408b388125cc200d2a0e0ccb71f4075071ff7e3f" exitCode=0 Sep 18 17:56:01.300661 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:01.300304 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" event={"ID":"452c133b-e9db-4a10-86dd-eabb11c3b7a6","Type":"ContainerDied","Data":"4b303367bbfa02ffe0ccc30f408b388125cc200d2a0e0ccb71f4075071ff7e3f"} Sep 18 17:56:02.442298 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.442267 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:56:02.504284 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.504252 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util\") pod \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " Sep 18 17:56:02.504421 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.504293 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle\") pod \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " Sep 18 17:56:02.504421 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.504329 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6mj\" (UniqueName: \"kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj\") pod \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\" (UID: \"452c133b-e9db-4a10-86dd-eabb11c3b7a6\") " Sep 18 17:56:02.505099 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.505068 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle" (OuterVolumeSpecName: "bundle") pod "452c133b-e9db-4a10-86dd-eabb11c3b7a6" (UID: "452c133b-e9db-4a10-86dd-eabb11c3b7a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:02.506491 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.506465 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj" (OuterVolumeSpecName: "kube-api-access-cf6mj") pod "452c133b-e9db-4a10-86dd-eabb11c3b7a6" (UID: "452c133b-e9db-4a10-86dd-eabb11c3b7a6"). InnerVolumeSpecName "kube-api-access-cf6mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:56:02.510008 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.509977 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util" (OuterVolumeSpecName: "util") pod "452c133b-e9db-4a10-86dd-eabb11c3b7a6" (UID: "452c133b-e9db-4a10-86dd-eabb11c3b7a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:02.605460 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.605393 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:02.605460 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.605419 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c133b-e9db-4a10-86dd-eabb11c3b7a6-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:02.605460 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:02.605428 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6mj\" (UniqueName: \"kubernetes.io/projected/452c133b-e9db-4a10-86dd-eabb11c3b7a6-kube-api-access-cf6mj\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:03.308965 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:03.308892 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" Sep 18 17:56:03.308965 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:03.308902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrzhnm" event={"ID":"452c133b-e9db-4a10-86dd-eabb11c3b7a6","Type":"ContainerDied","Data":"2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e"} Sep 18 17:56:03.308965 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:03.308945 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa51d411ebcae47cc456dd27016c640e00dc958bca8ad752e443a31659ee84e" Sep 18 17:56:11.782648 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782609 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq"] Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782866 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="extract" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782880 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="extract" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782887 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="util" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782893 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="util" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782903 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="pull" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782912 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="pull" Sep 18 17:56:11.783069 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.782959 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="452c133b-e9db-4a10-86dd-eabb11c3b7a6" containerName="extract" Sep 18 17:56:11.785622 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.785606 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.788573 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.788554 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q2zwl\"" Sep 18 17:56:11.788668 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.788556 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:56:11.789268 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.789245 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:56:11.812462 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.812437 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq"] Sep 18 17:56:11.868099 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.868068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.868259 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.868124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.868259 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.868201 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnjj\" (UniqueName: \"kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.969179 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.969145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.969320 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.969235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.969320 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.969262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnjj\" (UniqueName: \"kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.969552 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.969531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.969552 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.969546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:11.981905 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:11.981883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnjj\" (UniqueName: \"kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:12.094451 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:12.094373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:12.432991 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:12.432955 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq"] Sep 18 17:56:12.458278 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:12.458249 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173b6600_3723_4ed5_84b8_0af4d07e4216.slice/crio-8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a WatchSource:0}: Error finding container 8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a: Status 404 returned error can't find the container with id 8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a Sep 18 17:56:13.348015 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:13.347931 2570 generic.go:356] "Generic (PLEG): container finished" podID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerID="6c23bc206605e2c259a20434be6d2381fd9d739a0b85bc17b1dd1ac193b660a5" exitCode=0 Sep 18 17:56:13.348405 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:13.348013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" event={"ID":"173b6600-3723-4ed5-84b8-0af4d07e4216","Type":"ContainerDied","Data":"6c23bc206605e2c259a20434be6d2381fd9d739a0b85bc17b1dd1ac193b660a5"} Sep 18 17:56:13.348405 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:13.348064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" event={"ID":"173b6600-3723-4ed5-84b8-0af4d07e4216","Type":"ContainerStarted","Data":"8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a"} Sep 18 17:56:14.353942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.353876 2570 generic.go:356] "Generic (PLEG): container finished" podID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerID="6bea15e60286aa51c770c17673cf3a48bcc7002c3616dbea79d036e9e19e1221" exitCode=0 Sep 18 17:56:14.353942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.353923 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" event={"ID":"173b6600-3723-4ed5-84b8-0af4d07e4216","Type":"ContainerDied","Data":"6bea15e60286aa51c770c17673cf3a48bcc7002c3616dbea79d036e9e19e1221"} Sep 18 17:56:14.402747 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.402723 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc"] Sep 18 17:56:14.408093 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.408076 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.411455 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.411436 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 17:56:14.411655 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.411624 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 17:56:14.411905 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.411853 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6jc6c\"" Sep 18 17:56:14.411982 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.411928 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 17:56:14.412030 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.411853 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 17:56:14.449518 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.449481 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc"] Sep 18 17:56:14.490707 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.490671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.490836 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.490766 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.490836 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.490825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.490927 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.490865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.490927 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.490903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v22\" (UniqueName: \"kubernetes.io/projected/2f1f88f2-1183-46c2-a013-00aeedc21116-kube-api-access-85v22\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592039 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592161 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592161 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592311 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592311 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v22\" (UniqueName: \"kubernetes.io/projected/2f1f88f2-1183-46c2-a013-00aeedc21116-kube-api-access-85v22\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.592578 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.592544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.594108 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.594088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f1f88f2-1183-46c2-a013-00aeedc21116-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.594431 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.594416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.594569 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.594550 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f1f88f2-1183-46c2-a013-00aeedc21116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.602726 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.602705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v22\" (UniqueName: \"kubernetes.io/projected/2f1f88f2-1183-46c2-a013-00aeedc21116-kube-api-access-85v22\") pod \"opendatahub-operator-controller-manager-649dc7b766-xw2dc\" (UID: \"2f1f88f2-1183-46c2-a013-00aeedc21116\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:14.717506 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:14.717477 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:15.044286 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.044209 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc"] Sep 18 17:56:15.047105 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:15.047077 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1f88f2_1183_46c2_a013_00aeedc21116.slice/crio-5f88a85c9586974a3f411aa320d73c67f53904fd8e52144b6f13506b2ee40b86 WatchSource:0}: Error finding container 5f88a85c9586974a3f411aa320d73c67f53904fd8e52144b6f13506b2ee40b86: Status 404 returned error can't find the container with id 5f88a85c9586974a3f411aa320d73c67f53904fd8e52144b6f13506b2ee40b86 Sep 18 17:56:15.248442 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.248412 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597"] Sep 18 17:56:15.277862 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.277838 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.303601 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.303543 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:56:15.304796 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.304777 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 17:56:15.305275 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.305257 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597"] Sep 18 17:56:15.306426 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.306411 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 17:56:15.306674 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.306656 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jh2nx\"" Sep 18 17:56:15.306908 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.306897 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 17:56:15.308121 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.308107 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 18 17:56:15.359460 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.359432 2570 generic.go:356] "Generic (PLEG): container finished" podID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerID="9b478630bc2b333006942acdf9701bbbdea7073d0db2372a91cef3f49d680f45" exitCode=0 Sep 18 17:56:15.359843 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.359518 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" event={"ID":"173b6600-3723-4ed5-84b8-0af4d07e4216","Type":"ContainerDied","Data":"9b478630bc2b333006942acdf9701bbbdea7073d0db2372a91cef3f49d680f45"} Sep 18 17:56:15.360743 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.360720 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" event={"ID":"2f1f88f2-1183-46c2-a013-00aeedc21116","Type":"ContainerStarted","Data":"5f88a85c9586974a3f411aa320d73c67f53904fd8e52144b6f13506b2ee40b86"} Sep 18 17:56:15.397942 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.397920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8778s\" (UniqueName: \"kubernetes.io/projected/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-kube-api-access-8778s\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.398023 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.397947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.398023 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.397964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-manager-config\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.398023 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.397985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.498844 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.498818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8778s\" (UniqueName: \"kubernetes.io/projected/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-kube-api-access-8778s\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.498939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.498849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.498939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.498867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-manager-config\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.498939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.498889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.499628 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.499599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-manager-config\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.501283 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.501264 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.501385 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.501368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.513621 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.513598 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8778s\" (UniqueName: \"kubernetes.io/projected/669eecf6-1b5f-4b08-9b30-b40a64fb27ce-kube-api-access-8778s\") pod \"lws-controller-manager-786dfb84b9-ps597\" (UID: \"669eecf6-1b5f-4b08-9b30-b40a64fb27ce\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.586862 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.586795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:15.918735 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:15.918619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597"] Sep 18 17:56:15.921545 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:15.921515 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669eecf6_1b5f_4b08_9b30_b40a64fb27ce.slice/crio-fcee1798a51266b0b7615aa3f7c6faacad01f8d7b47652d583563bb4072e4529 WatchSource:0}: Error finding container fcee1798a51266b0b7615aa3f7c6faacad01f8d7b47652d583563bb4072e4529: Status 404 returned error can't find the container with id fcee1798a51266b0b7615aa3f7c6faacad01f8d7b47652d583563bb4072e4529 Sep 18 17:56:16.367574 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.367487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" event={"ID":"669eecf6-1b5f-4b08-9b30-b40a64fb27ce","Type":"ContainerStarted","Data":"fcee1798a51266b0b7615aa3f7c6faacad01f8d7b47652d583563bb4072e4529"} Sep 18 17:56:16.511041 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.511015 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:16.608089 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.608059 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle\") pod \"173b6600-3723-4ed5-84b8-0af4d07e4216\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " Sep 18 17:56:16.608476 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.608107 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwnjj\" (UniqueName: \"kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj\") pod \"173b6600-3723-4ed5-84b8-0af4d07e4216\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " Sep 18 17:56:16.608476 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.608165 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util\") pod \"173b6600-3723-4ed5-84b8-0af4d07e4216\" (UID: \"173b6600-3723-4ed5-84b8-0af4d07e4216\") " Sep 18 17:56:16.609080 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.609052 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle" (OuterVolumeSpecName: "bundle") pod "173b6600-3723-4ed5-84b8-0af4d07e4216" (UID: "173b6600-3723-4ed5-84b8-0af4d07e4216"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:16.611153 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.611113 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj" (OuterVolumeSpecName: "kube-api-access-zwnjj") pod "173b6600-3723-4ed5-84b8-0af4d07e4216" (UID: "173b6600-3723-4ed5-84b8-0af4d07e4216"). InnerVolumeSpecName "kube-api-access-zwnjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:56:16.616941 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.616899 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util" (OuterVolumeSpecName: "util") pod "173b6600-3723-4ed5-84b8-0af4d07e4216" (UID: "173b6600-3723-4ed5-84b8-0af4d07e4216"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:16.708915 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.708879 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:16.708915 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.708914 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnjj\" (UniqueName: \"kubernetes.io/projected/173b6600-3723-4ed5-84b8-0af4d07e4216-kube-api-access-zwnjj\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:16.709150 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:16.708929 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/173b6600-3723-4ed5-84b8-0af4d07e4216-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:17.375693 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:17.375661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" event={"ID":"173b6600-3723-4ed5-84b8-0af4d07e4216","Type":"ContainerDied","Data":"8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a"} Sep 18 17:56:17.376078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:17.375709 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2eec5e5dd871cafab41641908d2f9f4cfe38422f0f986830b3fc2a2317e09a" Sep 18 17:56:17.376078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:17.375676 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xd2nq" Sep 18 17:56:19.386539 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:19.386497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" event={"ID":"669eecf6-1b5f-4b08-9b30-b40a64fb27ce","Type":"ContainerStarted","Data":"d1f29003c9f22b1915591b9ebf904366b10789eb5ccd9965f35e98844cb767a9"} Sep 18 17:56:19.386982 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:19.386566 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:19.388134 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:19.388110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" event={"ID":"2f1f88f2-1183-46c2-a013-00aeedc21116","Type":"ContainerStarted","Data":"1edfb37c619d87de5499a98eeedbf05ec3ba78ece78a1f0503d419e0198bd5d3"} Sep 18 17:56:19.442109 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:19.442058 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" podStartSLOduration=1.532241393 podStartE2EDuration="4.442042007s" totalImagesPullingTime="2.909800614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:19.41169365 +0000 UTC m=+384.504348752" watchObservedRunningTime="2026-09-18 17:56:19.442042007 +0000 UTC m=+384.534697093" Sep 18 17:56:20.394236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:20.394114 2570 generic.go:356] "Generic (PLEG): container finished" podID="2f1f88f2-1183-46c2-a013-00aeedc21116" containerID="1edfb37c619d87de5499a98eeedbf05ec3ba78ece78a1f0503d419e0198bd5d3" exitCode=0 Sep 18 17:56:20.394236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:20.394168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" event={"ID":"2f1f88f2-1183-46c2-a013-00aeedc21116","Type":"ContainerDied","Data":"1edfb37c619d87de5499a98eeedbf05ec3ba78ece78a1f0503d419e0198bd5d3"} Sep 18 17:56:21.401496 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:21.401459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" event={"ID":"2f1f88f2-1183-46c2-a013-00aeedc21116","Type":"ContainerStarted","Data":"fc3ded5be1e709b163b25d8ac0b53a45c9cdad3b7092297d84641f586e9243e1"} Sep 18 17:56:21.401837 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:21.401630 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:30.396303 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:30.396270 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-ps597" Sep 18 17:56:30.416880 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:30.416830 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" podStartSLOduration=11.044614355 podStartE2EDuration="16.416814676s" totalImagesPullingTime="4.288083744s" totalInitContainerRuntime="1.084116577s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:21.545284491 +0000 UTC m=+386.637939599" watchObservedRunningTime="2026-09-18 17:56:30.416814676 +0000 UTC m=+395.509469763" Sep 18 17:56:32.407901 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:32.407873 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-xw2dc" Sep 18 17:56:44.215236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215165 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215449 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="extract" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215462 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="extract" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215477 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="pull" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215483 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="pull" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215494 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="util" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215499 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="util" Sep 18 17:56:44.215604 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.215541 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="173b6600-3723-4ed5-84b8-0af4d07e4216" containerName="extract" Sep 18 17:56:44.218321 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.218304 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.223408 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.223388 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 17:56:44.223895 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.223873 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 17:56:44.224903 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.224880 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 17:56:44.224984 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.224926 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6xc2d\"" Sep 18 17:56:44.240959 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.240937 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:44.309669 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309669 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309849 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309849 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309923 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309923 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.309923 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkjl\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.310037 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309924 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.310037 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.309961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.398062 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.398034 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll"] Sep 18 17:56:44.401008 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.400993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.410417 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410513 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410513 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410513 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410466 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkjl\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410513 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410513 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410751 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410751 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410751 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410901 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410952 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410907 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.410988 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.410963 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.411150 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.411128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.411216 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.411198 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.412748 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.412731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.413013 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.412996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.427427 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.427405 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll"] Sep 18 17:56:44.427925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.427901 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkjl\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.432996 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.432973 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.511080 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511080 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511080 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511055 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a091d1-e86e-4198-8507-bffcfa40906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a091d1-e86e-4198-8507-bffcfa40906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw98\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-kube-api-access-9rw98\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.511306 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.511225 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.528038 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.528011 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:44.611964 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.611935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612101 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.611980 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a091d1-e86e-4198-8507-bffcfa40906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612101 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.611996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612211 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw98\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-kube-api-access-9rw98\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612211 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612318 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612318 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612318 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a091d1-e86e-4198-8507-bffcfa40906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612318 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612511 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612423 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612566 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612636 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612621 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.612684 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.612663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.613140 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.613116 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a091d1-e86e-4198-8507-bffcfa40906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.614515 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.614490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a091d1-e86e-4198-8507-bffcfa40906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.614605 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.614589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a091d1-e86e-4198-8507-bffcfa40906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.641048 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.641023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw98\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-kube-api-access-9rw98\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.641364 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.641347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a091d1-e86e-4198-8507-bffcfa40906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll\" (UID: \"49a091d1-e86e-4198-8507-bffcfa40906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.710354 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.710317 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:44.831516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.831492 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:44.833260 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:44.833234 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93a8463_71ca_46a8_8f43_612ce09af221.slice/crio-976de58eaab2fd9f9c1ebd2b5343849456202df8674852ecf3fe2a889addefa6 WatchSource:0}: Error finding container 976de58eaab2fd9f9c1ebd2b5343849456202df8674852ecf3fe2a889addefa6: Status 404 returned error can't find the container with id 976de58eaab2fd9f9c1ebd2b5343849456202df8674852ecf3fe2a889addefa6 Sep 18 17:56:44.995066 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:44.995039 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll"] Sep 18 17:56:44.997463 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:44.997435 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a091d1_e86e_4198_8507_bffcfa40906d.slice/crio-7f0bca3d3eae6cd5e5dab3a4a5b8362b12b4e2def35900af8722250d52b5888f WatchSource:0}: Error finding container 7f0bca3d3eae6cd5e5dab3a4a5b8362b12b4e2def35900af8722250d52b5888f: Status 404 returned error can't find the container with id 7f0bca3d3eae6cd5e5dab3a4a5b8362b12b4e2def35900af8722250d52b5888f Sep 18 17:56:45.502532 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.502497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" event={"ID":"c93a8463-71ca-46a8-8f43-612ce09af221","Type":"ContainerStarted","Data":"976de58eaab2fd9f9c1ebd2b5343849456202df8674852ecf3fe2a889addefa6"} Sep 18 17:56:45.503724 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.503696 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" event={"ID":"49a091d1-e86e-4198-8507-bffcfa40906d","Type":"ContainerStarted","Data":"7f0bca3d3eae6cd5e5dab3a4a5b8362b12b4e2def35900af8722250d52b5888f"} Sep 18 17:56:45.896412 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.896315 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54g5v"] Sep 18 17:56:45.902452 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.901763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:45.905611 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.905384 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b472n\"" Sep 18 17:56:45.905611 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.905513 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 18 17:56:45.919703 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:45.919625 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54g5v"] Sep 18 17:56:46.026682 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.026372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tp2\" (UniqueName: \"kubernetes.io/projected/78abb58b-3458-4972-a656-d979f51c3b21-kube-api-access-26tp2\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.026682 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.026469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.127723 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.127496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tp2\" (UniqueName: \"kubernetes.io/projected/78abb58b-3458-4972-a656-d979f51c3b21-kube-api-access-26tp2\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.127723 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.127621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.127955 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.127843 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 18 17:56:46.127955 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.127918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert podName:78abb58b-3458-4972-a656-d979f51c3b21 nodeName:}" failed. No retries permitted until 2026-09-18 17:56:46.627897348 +0000 UTC m=+411.720552435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert") pod "odh-model-controller-7998c67dc4-54g5v" (UID: "78abb58b-3458-4972-a656-d979f51c3b21") : secret "odh-model-controller-webhook-cert" not found Sep 18 17:56:46.138130 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.138089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tp2\" (UniqueName: \"kubernetes.io/projected/78abb58b-3458-4972-a656-d979f51c3b21-kube-api-access-26tp2\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.632891 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.632848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:46.633382 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.633066 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 18 17:56:46.633382 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.633143 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert podName:78abb58b-3458-4972-a656-d979f51c3b21 nodeName:}" failed. No retries permitted until 2026-09-18 17:56:47.633122789 +0000 UTC m=+412.725777869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert") pod "odh-model-controller-7998c67dc4-54g5v" (UID: "78abb58b-3458-4972-a656-d979f51c3b21") : secret "odh-model-controller-webhook-cert" not found Sep 18 17:56:46.673525 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.673477 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x5t7d"] Sep 18 17:56:46.678369 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.678347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:46.681572 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.681375 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 18 17:56:46.699205 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.699118 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x5t7d"] Sep 18 17:56:46.835278 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.835078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:46.835603 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.835481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8srt\" (UniqueName: \"kubernetes.io/projected/f23a1124-405d-4273-83a3-cc0abd1a4db8-kube-api-access-d8srt\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:46.936287 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.936097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:46.936287 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.936178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8srt\" (UniqueName: \"kubernetes.io/projected/f23a1124-405d-4273-83a3-cc0abd1a4db8-kube-api-access-d8srt\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:46.936547 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.936309 2570 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 18 17:56:46.936547 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:46.936377 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs podName:f23a1124-405d-4273-83a3-cc0abd1a4db8 nodeName:}" failed. No retries permitted until 2026-09-18 17:56:47.436356895 +0000 UTC m=+412.529011982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs") pod "model-serving-api-54d89bd5fc-x5t7d" (UID: "f23a1124-405d-4273-83a3-cc0abd1a4db8") : secret "model-serving-api-tls" not found Sep 18 17:56:46.948370 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:46.948343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8srt\" (UniqueName: \"kubernetes.io/projected/f23a1124-405d-4273-83a3-cc0abd1a4db8-kube-api-access-d8srt\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:47.441331 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.441291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:47.443786 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.443752 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23a1124-405d-4273-83a3-cc0abd1a4db8-tls-certs\") pod \"model-serving-api-54d89bd5fc-x5t7d\" (UID: \"f23a1124-405d-4273-83a3-cc0abd1a4db8\") " pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:47.593077 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.593019 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:47.642521 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.642478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:47.645232 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.645200 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78abb58b-3458-4972-a656-d979f51c3b21-cert\") pod \"odh-model-controller-7998c67dc4-54g5v\" (UID: \"78abb58b-3458-4972-a656-d979f51c3b21\") " pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:47.722277 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.722163 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:47.924653 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.924571 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:47.924778 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.924670 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:47.924778 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.924702 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:47.930762 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.930727 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:47.930831 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.930799 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:47.931506 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:47.930842 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:48.114300 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.114136 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54g5v"] Sep 18 17:56:48.117903 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:48.117862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78abb58b_3458_4972_a656_d979f51c3b21.slice/crio-349caf34e81bc1d7dee56d9353c476fee52f16381c1f9ce668db442b1daff28d WatchSource:0}: Error finding container 349caf34e81bc1d7dee56d9353c476fee52f16381c1f9ce668db442b1daff28d: Status 404 returned error can't find the container with id 349caf34e81bc1d7dee56d9353c476fee52f16381c1f9ce668db442b1daff28d Sep 18 17:56:48.118258 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.118179 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x5t7d"] Sep 18 17:56:48.120240 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:48.120173 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23a1124_405d_4273_83a3_cc0abd1a4db8.slice/crio-896debae30afcdf25cb09621122d0122ae226f2b5f0ba3e80cb48cb91d7d4008 WatchSource:0}: Error finding container 896debae30afcdf25cb09621122d0122ae226f2b5f0ba3e80cb48cb91d7d4008: Status 404 returned error can't find the container with id 896debae30afcdf25cb09621122d0122ae226f2b5f0ba3e80cb48cb91d7d4008 Sep 18 17:56:48.521629 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.521589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" event={"ID":"c93a8463-71ca-46a8-8f43-612ce09af221","Type":"ContainerStarted","Data":"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d"} Sep 18 17:56:48.522838 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.522811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" event={"ID":"f23a1124-405d-4273-83a3-cc0abd1a4db8","Type":"ContainerStarted","Data":"896debae30afcdf25cb09621122d0122ae226f2b5f0ba3e80cb48cb91d7d4008"} Sep 18 17:56:48.523976 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.523946 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" event={"ID":"78abb58b-3458-4972-a656-d979f51c3b21","Type":"ContainerStarted","Data":"349caf34e81bc1d7dee56d9353c476fee52f16381c1f9ce668db442b1daff28d"} Sep 18 17:56:48.525482 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.525463 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" event={"ID":"49a091d1-e86e-4198-8507-bffcfa40906d","Type":"ContainerStarted","Data":"eeae8309a9e0ca8aa2ed6489000957d293ce5b7e48e365e319e44cf56013e29c"} Sep 18 17:56:48.528992 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.528973 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:48.546656 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.546615 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" podStartSLOduration=1.4513744499999999 podStartE2EDuration="4.546601823s" totalImagesPullingTime="3.095227373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:48.544686694 +0000 UTC m=+413.637341797" watchObservedRunningTime="2026-09-18 17:56:48.546601823 +0000 UTC m=+413.639256911" Sep 18 17:56:48.573559 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.573517 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" podStartSLOduration=1.648808775 podStartE2EDuration="4.573498037s" totalImagesPullingTime="2.924689262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:48.571954146 +0000 UTC m=+413.664609262" watchObservedRunningTime="2026-09-18 17:56:48.573498037 +0000 UTC m=+413.666153123" Sep 18 17:56:48.711154 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:48.711104 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:49.531235 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:49.530375 2570 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Sep 18 17:56:49.531235 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:49.530443 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Sep 18 17:56:49.717043 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:49.716997 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:50.529227 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:50.529168 2570 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Sep 18 17:56:50.529382 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:50.529256 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Sep 18 17:56:50.538587 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:50.538560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:50.539553 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:50.539528 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll" Sep 18 17:56:50.629805 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:50.629756 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:51.529077 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:51.529037 2570 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Sep 18 17:56:51.529462 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:51.529099 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Sep 18 17:56:51.542875 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:51.542835 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" containerID="cri-o://890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d" gracePeriod=30 Sep 18 17:56:52.548540 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:52.548510 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" event={"ID":"f23a1124-405d-4273-83a3-cc0abd1a4db8","Type":"ContainerStarted","Data":"0fa6496fe0c3ab93a7d546ff50f01aafed2d9edefe3d8c84f8e45a46aa749bd2"} Sep 18 17:56:52.564276 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:52.564245 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:56:52.598687 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:52.598635 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" podStartSLOduration=2.5156469870000002 podStartE2EDuration="6.598615493s" totalImagesPullingTime="4.082968506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:52.596130804 +0000 UTC m=+417.688785906" watchObservedRunningTime="2026-09-18 17:56:52.598615493 +0000 UTC m=+417.691270582" Sep 18 17:56:53.555085 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:53.555046 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" event={"ID":"78abb58b-3458-4972-a656-d979f51c3b21","Type":"ContainerStarted","Data":"3c02887ac6ea921261ffc930c6ce9fd8a0926b620e108ceddbb932aeeb8b256f"} Sep 18 17:56:53.555477 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:53.555142 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:56:53.582170 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:53.582126 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" podStartSLOduration=4.159748386 podStartE2EDuration="8.582112644s" totalImagesPullingTime="4.422364258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:53.579299745 +0000 UTC m=+418.671954846" watchObservedRunningTime="2026-09-18 17:56:53.582112644 +0000 UTC m=+418.674767724" Sep 18 17:56:54.659711 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.659675 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c"] Sep 18 17:56:54.661950 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.661931 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.676648 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.676625 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 17:56:54.676774 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.676757 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 17:56:54.690425 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.690402 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m2bc8\"" Sep 18 17:56:54.700627 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.700608 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 17:56:54.704904 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.704879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/124371a9-200d-4575-aeb7-314b79294334-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705082 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705259 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705365 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705423 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xzgs\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-kube-api-access-8xzgs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705472 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.705472 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.705459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/124371a9-200d-4575-aeb7-314b79294334-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.707525 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.707506 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c"] Sep 18 17:56:54.805949 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.805919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806088 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.805988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xzgs\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-kube-api-access-8xzgs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806088 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806088 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/124371a9-200d-4575-aeb7-314b79294334-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806379 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/124371a9-200d-4575-aeb7-314b79294334-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806379 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.806379 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.807016 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.806987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/124371a9-200d-4575-aeb7-314b79294334-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.808346 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.808314 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/124371a9-200d-4575-aeb7-314b79294334-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.808460 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.808444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.808507 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.808463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.808851 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.808835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/124371a9-200d-4575-aeb7-314b79294334-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.828561 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.828537 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xzgs\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-kube-api-access-8xzgs\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.839378 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.839353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124371a9-200d-4575-aeb7-314b79294334-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bpw6c\" (UID: \"124371a9-200d-4575-aeb7-314b79294334\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:54.969896 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:54.969815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:55.259492 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:55.259462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c"] Sep 18 17:56:55.260490 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:56:55.260465 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124371a9_200d_4575_aeb7_314b79294334.slice/crio-7171efe5ba30abf290a8bab381273865d21c9f015a96dc773dfef30c9b003d57 WatchSource:0}: Error finding container 7171efe5ba30abf290a8bab381273865d21c9f015a96dc773dfef30c9b003d57: Status 404 returned error can't find the container with id 7171efe5ba30abf290a8bab381273865d21c9f015a96dc773dfef30c9b003d57 Sep 18 17:56:55.564649 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:55.564571 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" event={"ID":"124371a9-200d-4575-aeb7-314b79294334","Type":"ContainerStarted","Data":"7171efe5ba30abf290a8bab381273865d21c9f015a96dc773dfef30c9b003d57"} Sep 18 17:56:56.815087 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.815017 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:56.820435 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820416 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820529 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820463 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820529 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820487 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820619 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820544 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqkjl\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820619 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820704 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820685 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820807 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820681 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:56.820807 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820722 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820920 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820781 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.820920 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820883 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:56.820920 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.820912 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:56.821097 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821075 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data\") pod \"c93a8463-71ca-46a8-8f43-612ce09af221\" (UID: \"c93a8463-71ca-46a8-8f43-612ce09af221\") " Sep 18 17:56:56.821237 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821220 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:56:56.821516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821418 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data" (OuterVolumeSpecName: "istio-data") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:56.821516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821480 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.821516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821494 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.821516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821510 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93a8463-71ca-46a8-8f43-612ce09af221-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.821516 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.821519 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.823143 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.823107 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:56:56.823262 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.823141 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl" (OuterVolumeSpecName: "kube-api-access-zqkjl") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "kube-api-access-zqkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:56:56.823324 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.823279 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:56:56.823582 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.823565 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token" (OuterVolumeSpecName: "istio-token") pod "c93a8463-71ca-46a8-8f43-612ce09af221" (UID: "c93a8463-71ca-46a8-8f43-612ce09af221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:56:56.922445 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.922406 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93a8463-71ca-46a8-8f43-612ce09af221-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.922445 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.922443 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.922677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.922458 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkjl\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-kube-api-access-zqkjl\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.922677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.922471 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93a8463-71ca-46a8-8f43-612ce09af221-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:56.922677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:56.922483 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93a8463-71ca-46a8-8f43-612ce09af221-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:56:57.576144 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.576097 2570 generic.go:356] "Generic (PLEG): container finished" podID="c93a8463-71ca-46a8-8f43-612ce09af221" containerID="890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d" exitCode=0 Sep 18 17:56:57.576358 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.576167 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" Sep 18 17:56:57.576358 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.576200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" event={"ID":"c93a8463-71ca-46a8-8f43-612ce09af221","Type":"ContainerDied","Data":"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d"} Sep 18 17:56:57.576358 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.576258 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf" event={"ID":"c93a8463-71ca-46a8-8f43-612ce09af221","Type":"ContainerDied","Data":"976de58eaab2fd9f9c1ebd2b5343849456202df8674852ecf3fe2a889addefa6"} Sep 18 17:56:57.576358 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.576281 2570 scope.go:117] "RemoveContainer" containerID="890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d" Sep 18 17:56:57.614478 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.614442 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:57.621539 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.621512 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc4fhjf"] Sep 18 17:56:57.728033 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.727866 2570 scope.go:117] "RemoveContainer" containerID="890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d" Sep 18 17:56:57.728297 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:56:57.728263 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d\": container with ID starting with 890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d not found: ID does not exist" containerID="890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d" Sep 18 17:56:57.728387 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.728311 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d"} err="failed to get container status \"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d\": rpc error: code = NotFound desc = could not find container \"890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d\": container with ID starting with 890379a8c92fdd78c931f65301b2ed96537b1b97ed7ebc577ab14b313479419d not found: ID does not exist" Sep 18 17:56:57.756960 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.756926 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:57.757045 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:57.756997 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:56:58.582118 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.582077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" event={"ID":"124371a9-200d-4575-aeb7-314b79294334","Type":"ContainerStarted","Data":"0e0b6d96a43d3d782a6815f1bdb8b47c54864d716c71769672d0fee922f1e026"} Sep 18 17:56:58.582575 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.582148 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:56:58.583971 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.583947 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-bpw6c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 17:56:58.584096 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.583993 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" podUID="124371a9-200d-4575-aeb7-314b79294334" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 17:56:58.741689 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.741633 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" podStartSLOduration=2.247485025 podStartE2EDuration="4.741613122s" totalImagesPullingTime="2.494128097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:58.736745187 +0000 UTC m=+423.829400295" watchObservedRunningTime="2026-09-18 17:56:58.741613122 +0000 UTC m=+423.834268209" Sep 18 17:56:58.941746 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:58.941714 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" path="/var/lib/kubelet/pods/c93a8463-71ca-46a8-8f43-612ce09af221/volumes" Sep 18 17:56:59.588256 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:56:59.588229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bpw6c" Sep 18 17:57:03.559370 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:03.559342 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x5t7d" Sep 18 17:57:04.562301 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:04.562272 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-54g5v" Sep 18 17:57:52.392669 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.392628 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp"] Sep 18 17:57:52.393097 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.392890 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" Sep 18 17:57:52.393097 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.392901 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" Sep 18 17:57:52.393097 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.392956 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93a8463-71ca-46a8-8f43-612ce09af221" containerName="istio-proxy" Sep 18 17:57:52.394860 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.394840 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.398436 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.398416 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q2zwl\"" Sep 18 17:57:52.398839 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.398818 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:57:52.399139 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.399113 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:57:52.408740 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.408716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp"] Sep 18 17:57:52.444541 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.444515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhs5f\" (UniqueName: \"kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.444677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.444547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.444677 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.444565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.513687 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.513655 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc"] Sep 18 17:57:52.515894 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.515878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.536013 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.535984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc"] Sep 18 17:57:52.544989 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.544964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.545083 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.544999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.545083 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.545168 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqzf\" (UniqueName: \"kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.545252 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhs5f\" (UniqueName: \"kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.545294 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.545505 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545477 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.545543 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.545512 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.565051 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.565019 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhs5f\" (UniqueName: \"kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.641369 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.641346 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79"] Sep 18 17:57:52.643737 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.643702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.646084 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.646084 ip-10-0-133-72 kubenswrapper[2570]: E0918 17:57:52.646070 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" is forbidden: User \"system:node:ip-10-0-133-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-133-72.ec2.internal' and this object" podUID="82b86510-51bb-49ba-b12c-05855f15a047" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.646285 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.646285 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646138 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.646285 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snc2q\" (UniqueName: \"kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.646285 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.646442 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqzf\" (UniqueName: \"kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.646495 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646476 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.646535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.646494 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.667889 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.667871 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqzf\" (UniqueName: \"kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.670202 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.670164 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79"] Sep 18 17:57:52.704031 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.704010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:52.747609 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.747576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.747845 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.747822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snc2q\" (UniqueName: \"kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.748005 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.747986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.748427 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.748000 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.748554 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.748273 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.749429 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.749391 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x"] Sep 18 17:57:52.754098 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.754076 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.768157 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.768101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snc2q\" (UniqueName: \"kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.770582 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.770520 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x"] Sep 18 17:57:52.823761 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.823729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:52.850117 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.850090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcdx\" (UniqueName: \"kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.850334 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.850137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.850334 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.850225 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951474 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcdx\" (UniqueName: \"kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951632 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951632 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951861 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951935 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951865 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.951973 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.951941 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:52.961721 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.961694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcdx\" (UniqueName: \"kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:52.986538 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:52.986505 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp"] Sep 18 17:57:52.990531 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:57:52.990503 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaaab3b_716f_48bd_80b5_283fe89d1540.slice/crio-d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f WatchSource:0}: Error finding container d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f: Status 404 returned error can't find the container with id d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f Sep 18 17:57:53.068800 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.068775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:53.100749 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.100723 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc"] Sep 18 17:57:53.146113 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:57:53.146084 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff27718_294b_451b_9783_311e5c0d1f60.slice/crio-5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9 WatchSource:0}: Error finding container 5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9: Status 404 returned error can't find the container with id 5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9 Sep 18 17:57:53.232837 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.232807 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79"] Sep 18 17:57:53.234105 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:57:53.234073 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b86510_51bb_49ba_b12c_05855f15a047.slice/crio-7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108 WatchSource:0}: Error finding container 7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108: Status 404 returned error can't find the container with id 7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108 Sep 18 17:57:53.347940 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.347904 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x"] Sep 18 17:57:53.349727 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:57:53.349694 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ee2b10_d09e_4db1_850b_c16dbd21dcdd.slice/crio-ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1 WatchSource:0}: Error finding container ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1: Status 404 returned error can't find the container with id ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1 Sep 18 17:57:53.822551 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.822471 2570 generic.go:356] "Generic (PLEG): container finished" podID="cff27718-294b-451b-9783-311e5c0d1f60" containerID="6cec3cf9ca40b3d57b76a9744fb227db3a112775745249700d899d6adc3c65ef" exitCode=0 Sep 18 17:57:53.822939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.822556 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" event={"ID":"cff27718-294b-451b-9783-311e5c0d1f60","Type":"ContainerDied","Data":"6cec3cf9ca40b3d57b76a9744fb227db3a112775745249700d899d6adc3c65ef"} Sep 18 17:57:53.822939 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.822607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" event={"ID":"cff27718-294b-451b-9783-311e5c0d1f60","Type":"ContainerStarted","Data":"5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9"} Sep 18 17:57:53.824341 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.824308 2570 generic.go:356] "Generic (PLEG): container finished" podID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerID="22dcdee721a60d1ee40b21074f31cd7cf4d39a2c6271be34a0979fa341bfac88" exitCode=0 Sep 18 17:57:53.824461 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.824376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" event={"ID":"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd","Type":"ContainerDied","Data":"22dcdee721a60d1ee40b21074f31cd7cf4d39a2c6271be34a0979fa341bfac88"} Sep 18 17:57:53.824461 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.824419 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" event={"ID":"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd","Type":"ContainerStarted","Data":"ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1"} Sep 18 17:57:53.826052 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.826029 2570 generic.go:356] "Generic (PLEG): container finished" podID="82b86510-51bb-49ba-b12c-05855f15a047" containerID="f778623eb2543beccf69dd16e7feef0ea084720d08ab7d01a7c70742dcdcd741" exitCode=0 Sep 18 17:57:53.826145 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.826112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" event={"ID":"82b86510-51bb-49ba-b12c-05855f15a047","Type":"ContainerDied","Data":"f778623eb2543beccf69dd16e7feef0ea084720d08ab7d01a7c70742dcdcd741"} Sep 18 17:57:53.826232 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.826157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" event={"ID":"82b86510-51bb-49ba-b12c-05855f15a047","Type":"ContainerStarted","Data":"7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108"} Sep 18 17:57:53.827842 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.827822 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerID="5b6768436c3d86386a908e5d6c9b06c679bf70c349e5ee0674fa2ab7568347bd" exitCode=0 Sep 18 17:57:53.827941 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.827847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" event={"ID":"ccaaab3b-716f-48bd-80b5-283fe89d1540","Type":"ContainerDied","Data":"5b6768436c3d86386a908e5d6c9b06c679bf70c349e5ee0674fa2ab7568347bd"} Sep 18 17:57:53.827941 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:53.827888 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" event={"ID":"ccaaab3b-716f-48bd-80b5-283fe89d1540","Type":"ContainerStarted","Data":"d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f"} Sep 18 17:57:55.839432 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.839389 2570 generic.go:356] "Generic (PLEG): container finished" podID="cff27718-294b-451b-9783-311e5c0d1f60" containerID="50be5a6e5a8111243aa3cca9fb3f42f7d0af6003da603b2af582ba08adcd3c49" exitCode=0 Sep 18 17:57:55.839849 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.839477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" event={"ID":"cff27718-294b-451b-9783-311e5c0d1f60","Type":"ContainerDied","Data":"50be5a6e5a8111243aa3cca9fb3f42f7d0af6003da603b2af582ba08adcd3c49"} Sep 18 17:57:55.841370 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.841349 2570 generic.go:356] "Generic (PLEG): container finished" podID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerID="99d4cb5c58914e3b418afea378cf493342a4252d8a7b640c3b58b0789cdda3de" exitCode=0 Sep 18 17:57:55.841475 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.841424 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" event={"ID":"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd","Type":"ContainerDied","Data":"99d4cb5c58914e3b418afea378cf493342a4252d8a7b640c3b58b0789cdda3de"} Sep 18 17:57:55.843249 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.843230 2570 generic.go:356] "Generic (PLEG): container finished" podID="82b86510-51bb-49ba-b12c-05855f15a047" containerID="72f863c9a81e139ac4fe279d6f671d35d248f8319cc5e36490facbc84a02f1d0" exitCode=0 Sep 18 17:57:55.843315 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.843294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" event={"ID":"82b86510-51bb-49ba-b12c-05855f15a047","Type":"ContainerDied","Data":"72f863c9a81e139ac4fe279d6f671d35d248f8319cc5e36490facbc84a02f1d0"} Sep 18 17:57:55.845264 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.845243 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerID="5dc3efb99809a814578fbce536b57b3a59d29490965bc34401a13daf94de5973" exitCode=0 Sep 18 17:57:55.845355 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:55.845287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" event={"ID":"ccaaab3b-716f-48bd-80b5-283fe89d1540","Type":"ContainerDied","Data":"5dc3efb99809a814578fbce536b57b3a59d29490965bc34401a13daf94de5973"} Sep 18 17:57:56.852535 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.852505 2570 generic.go:356] "Generic (PLEG): container finished" podID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerID="fc04e021ebab2199024792afc5c0ff9dcab33859c83cf7f2026b7372ad47c6fa" exitCode=0 Sep 18 17:57:56.852925 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.852587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" event={"ID":"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd","Type":"ContainerDied","Data":"fc04e021ebab2199024792afc5c0ff9dcab33859c83cf7f2026b7372ad47c6fa"} Sep 18 17:57:56.854642 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.854616 2570 generic.go:356] "Generic (PLEG): container finished" podID="82b86510-51bb-49ba-b12c-05855f15a047" containerID="f354213f3eb99770943cbd9884b273726283d28cc1da4e09634f1c952c950f5b" exitCode=0 Sep 18 17:57:56.854774 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.854685 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" event={"ID":"82b86510-51bb-49ba-b12c-05855f15a047","Type":"ContainerDied","Data":"f354213f3eb99770943cbd9884b273726283d28cc1da4e09634f1c952c950f5b"} Sep 18 17:57:56.856741 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.856719 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerID="6631b190a6120f8ab7cc02cda4999e912fbbccb6132eef145768347867fbfcf2" exitCode=0 Sep 18 17:57:56.856862 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.856798 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" event={"ID":"ccaaab3b-716f-48bd-80b5-283fe89d1540","Type":"ContainerDied","Data":"6631b190a6120f8ab7cc02cda4999e912fbbccb6132eef145768347867fbfcf2"} Sep 18 17:57:56.858825 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.858801 2570 generic.go:356] "Generic (PLEG): container finished" podID="cff27718-294b-451b-9783-311e5c0d1f60" containerID="8ac5877953a1e22a382f30fd2e149d1dc3ee388cf804e2787778e89f7d5f7b46" exitCode=0 Sep 18 17:57:56.858934 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:56.858842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" event={"ID":"cff27718-294b-451b-9783-311e5c0d1f60","Type":"ContainerDied","Data":"8ac5877953a1e22a382f30fd2e149d1dc3ee388cf804e2787778e89f7d5f7b46"} Sep 18 17:57:57.996033 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:57.996009 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:58.039926 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.039898 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:58.073707 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.073680 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:58.076778 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.076757 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:58.090599 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snc2q\" (UniqueName: \"kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q\") pod \"82b86510-51bb-49ba-b12c-05855f15a047\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " Sep 18 17:57:58.090660 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util\") pod \"82b86510-51bb-49ba-b12c-05855f15a047\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " Sep 18 17:57:58.090660 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090642 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wqzf\" (UniqueName: \"kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf\") pod \"cff27718-294b-451b-9783-311e5c0d1f60\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " Sep 18 17:57:58.090742 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090673 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle\") pod \"82b86510-51bb-49ba-b12c-05855f15a047\" (UID: \"82b86510-51bb-49ba-b12c-05855f15a047\") " Sep 18 17:57:58.090742 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090699 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcdx\" (UniqueName: \"kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx\") pod \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " Sep 18 17:57:58.090820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090742 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle\") pod \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " Sep 18 17:57:58.090820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090771 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util\") pod \"ccaaab3b-716f-48bd-80b5-283fe89d1540\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " Sep 18 17:57:58.090820 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090794 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle\") pod \"ccaaab3b-716f-48bd-80b5-283fe89d1540\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " Sep 18 17:57:58.090961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util\") pod \"cff27718-294b-451b-9783-311e5c0d1f60\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " Sep 18 17:57:58.090961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090881 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle\") pod \"cff27718-294b-451b-9783-311e5c0d1f60\" (UID: \"cff27718-294b-451b-9783-311e5c0d1f60\") " Sep 18 17:57:58.090961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090906 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util\") pod \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\" (UID: \"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd\") " Sep 18 17:57:58.090961 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.090942 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhs5f\" (UniqueName: \"kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f\") pod \"ccaaab3b-716f-48bd-80b5-283fe89d1540\" (UID: \"ccaaab3b-716f-48bd-80b5-283fe89d1540\") " Sep 18 17:57:58.094316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.091470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle" (OuterVolumeSpecName: "bundle") pod "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" (UID: "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.094316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.091692 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle" (OuterVolumeSpecName: "bundle") pod "82b86510-51bb-49ba-b12c-05855f15a047" (UID: "82b86510-51bb-49ba-b12c-05855f15a047"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.094316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.092439 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle" (OuterVolumeSpecName: "bundle") pod "cff27718-294b-451b-9783-311e5c0d1f60" (UID: "cff27718-294b-451b-9783-311e5c0d1f60"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.094316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.094113 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q" (OuterVolumeSpecName: "kube-api-access-snc2q") pod "82b86510-51bb-49ba-b12c-05855f15a047" (UID: "82b86510-51bb-49ba-b12c-05855f15a047"). InnerVolumeSpecName "kube-api-access-snc2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:57:58.094316 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.094139 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f" (OuterVolumeSpecName: "kube-api-access-fhs5f") pod "ccaaab3b-716f-48bd-80b5-283fe89d1540" (UID: "ccaaab3b-716f-48bd-80b5-283fe89d1540"). InnerVolumeSpecName "kube-api-access-fhs5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:57:58.094684 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.094662 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle" (OuterVolumeSpecName: "bundle") pod "ccaaab3b-716f-48bd-80b5-283fe89d1540" (UID: "ccaaab3b-716f-48bd-80b5-283fe89d1540"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.095003 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.094940 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf" (OuterVolumeSpecName: "kube-api-access-8wqzf") pod "cff27718-294b-451b-9783-311e5c0d1f60" (UID: "cff27718-294b-451b-9783-311e5c0d1f60"). InnerVolumeSpecName "kube-api-access-8wqzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:57:58.095003 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.094980 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx" (OuterVolumeSpecName: "kube-api-access-rdcdx") pod "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" (UID: "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd"). InnerVolumeSpecName "kube-api-access-rdcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:57:58.099815 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.099792 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util" (OuterVolumeSpecName: "util") pod "82b86510-51bb-49ba-b12c-05855f15a047" (UID: "82b86510-51bb-49ba-b12c-05855f15a047"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.100362 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.100336 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util" (OuterVolumeSpecName: "util") pod "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" (UID: "e8ee2b10-d09e-4db1-850b-c16dbd21dcdd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.101364 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.101340 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util" (OuterVolumeSpecName: "util") pod "ccaaab3b-716f-48bd-80b5-283fe89d1540" (UID: "ccaaab3b-716f-48bd-80b5-283fe89d1540"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.101456 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.101437 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util" (OuterVolumeSpecName: "util") pod "cff27718-294b-451b-9783-311e5c0d1f60" (UID: "cff27718-294b-451b-9783-311e5c0d1f60"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:57:58.191615 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191578 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191615 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191610 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191615 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191619 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccaaab3b-716f-48bd-80b5-283fe89d1540-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191627 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191634 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff27718-294b-451b-9783-311e5c0d1f60-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191641 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191651 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhs5f\" (UniqueName: \"kubernetes.io/projected/ccaaab3b-716f-48bd-80b5-283fe89d1540-kube-api-access-fhs5f\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191668 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snc2q\" (UniqueName: \"kubernetes.io/projected/82b86510-51bb-49ba-b12c-05855f15a047-kube-api-access-snc2q\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191680 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-util\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191687 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqzf\" (UniqueName: \"kubernetes.io/projected/cff27718-294b-451b-9783-311e5c0d1f60-kube-api-access-8wqzf\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191695 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b86510-51bb-49ba-b12c-05855f15a047-bundle\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.191823 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.191702 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcdx\" (UniqueName: \"kubernetes.io/projected/e8ee2b10-d09e-4db1-850b-c16dbd21dcdd-kube-api-access-rdcdx\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 17:57:58.870603 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.870566 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" event={"ID":"cff27718-294b-451b-9783-311e5c0d1f60","Type":"ContainerDied","Data":"5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9"} Sep 18 17:57:58.870794 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.870620 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5462f6c1d51fdcc99e770735cb6d4e9fb238626988a0d47a3f22ba022ba3ecd9" Sep 18 17:57:58.870794 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.870561 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adk6cc" Sep 18 17:57:58.872497 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.872462 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" Sep 18 17:57:58.872625 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.872461 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z926x" event={"ID":"e8ee2b10-d09e-4db1-850b-c16dbd21dcdd","Type":"ContainerDied","Data":"ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1"} Sep 18 17:57:58.872625 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.872582 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab317b936349e9e5635e8f305ab9670f327122754b221f860e61c5606f3dacd1" Sep 18 17:57:58.874497 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.874462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" event={"ID":"82b86510-51bb-49ba-b12c-05855f15a047","Type":"ContainerDied","Data":"7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108"} Sep 18 17:57:58.874635 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.874501 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0080bd13c2c3c91b3afe7a686761553cb2a30e2041158bf8c23af654c63108" Sep 18 17:57:58.874635 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.874465 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvj79" Sep 18 17:57:58.876617 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.876595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" event={"ID":"ccaaab3b-716f-48bd-80b5-283fe89d1540","Type":"ContainerDied","Data":"d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f"} Sep 18 17:57:58.876725 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.876631 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbxwnp" Sep 18 17:57:58.876725 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:57:58.876634 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d060c92fc10d9e2ca4a9481c946d8fe043a386fb9b068e2d49ce285db201059f" Sep 18 17:58:59.241533 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241440 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7"] Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241727 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241740 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241753 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241760 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241769 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241774 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241783 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241788 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241793 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241798 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241805 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241810 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241818 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241825 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="util" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241835 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241840 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241847 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241852 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241858 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241863 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="pull" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241868 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241872 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241879 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241884 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241933 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b86510-51bb-49ba-b12c-05855f15a047" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241942 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccaaab3b-716f-48bd-80b5-283fe89d1540" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241948 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff27718-294b-451b-9783-311e5c0d1f60" containerName="extract" Sep 18 17:58:59.242078 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.241955 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ee2b10-d09e-4db1-850b-c16dbd21dcdd" containerName="extract" Sep 18 17:58:59.244944 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.244921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.248236 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.248217 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pnczq\"" Sep 18 17:58:59.264014 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.263992 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7"] Sep 18 17:58:59.355180 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355333 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355205 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355333 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355333 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355437 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxfs\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-kube-api-access-7rxfs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355437 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ca5410-7955-4a3d-8890-f90cb83ce783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355437 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355530 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.355530 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.355480 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456547 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456549 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456768 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxfs\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-kube-api-access-7rxfs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456768 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ca5410-7955-4a3d-8890-f90cb83ce783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456768 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456768 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456768 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456996 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456996 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.456996 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.456981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.457153 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.457003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.457153 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.457098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.457307 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.457285 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.457595 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.457571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ca5410-7955-4a3d-8890-f90cb83ce783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.458931 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.458910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.459249 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.459231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.472825 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.472799 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.473005 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.472988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxfs\" (UniqueName: \"kubernetes.io/projected/39ca5410-7955-4a3d-8890-f90cb83ce783-kube-api-access-7rxfs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-62zx7\" (UID: \"39ca5410-7955-4a3d-8890-f90cb83ce783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.554738 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.554677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:58:59.846754 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.846692 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7"] Sep 18 17:58:59.849086 ip-10-0-133-72 kubenswrapper[2570]: W0918 17:58:59.849042 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ca5410_7955_4a3d_8890_f90cb83ce783.slice/crio-26926dd356a0af9df28b5148945ead6524460cb6d15dfd266b7c120cdda9ec0f WatchSource:0}: Error finding container 26926dd356a0af9df28b5148945ead6524460cb6d15dfd266b7c120cdda9ec0f: Status 404 returned error can't find the container with id 26926dd356a0af9df28b5148945ead6524460cb6d15dfd266b7c120cdda9ec0f Sep 18 17:58:59.851102 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.851070 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:58:59.851209 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.851130 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:58:59.851209 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:58:59.851157 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 17:59:00.159448 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:00.159413 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" event={"ID":"39ca5410-7955-4a3d-8890-f90cb83ce783","Type":"ContainerStarted","Data":"395fd44d2ceefe252d8bbdd38ab22830829fc6401f15b9930a8c18236adae3d0"} Sep 18 17:59:00.159627 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:00.159466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" event={"ID":"39ca5410-7955-4a3d-8890-f90cb83ce783","Type":"ContainerStarted","Data":"26926dd356a0af9df28b5148945ead6524460cb6d15dfd266b7c120cdda9ec0f"} Sep 18 17:59:00.188114 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:00.188063 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podStartSLOduration=1.188047191 podStartE2EDuration="1.188047191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:00.185640714 +0000 UTC m=+545.278295817" watchObservedRunningTime="2026-09-18 17:59:00.188047191 +0000 UTC m=+545.280702277" Sep 18 17:59:00.555475 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:00.555390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:59:00.559841 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:00.559816 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:59:01.164250 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:01.164217 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 17:59:01.165150 ip-10-0-133-72 kubenswrapper[2570]: I0918 17:59:01.165134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 18:00:23.546462 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.546426 2570 generic.go:356] "Generic (PLEG): container finished" podID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerID="395fd44d2ceefe252d8bbdd38ab22830829fc6401f15b9930a8c18236adae3d0" exitCode=0 Sep 18 18:00:23.546860 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.546498 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" event={"ID":"39ca5410-7955-4a3d-8890-f90cb83ce783","Type":"ContainerDied","Data":"395fd44d2ceefe252d8bbdd38ab22830829fc6401f15b9930a8c18236adae3d0"} Sep 18 18:00:23.546860 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.546848 2570 scope.go:117] "RemoveContainer" containerID="395fd44d2ceefe252d8bbdd38ab22830829fc6401f15b9930a8c18236adae3d0" Sep 18 18:00:23.547792 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.547764 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:00:23.547898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.547846 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:00:23.547898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:23.547892 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:00:24.553541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:24.553500 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" event={"ID":"39ca5410-7955-4a3d-8890-f90cb83ce783","Type":"ContainerStarted","Data":"3310f3b2e7d7bc8f6a6ac415bd77c16a84ad6e9598d66a592016cacd83793e9a"} Sep 18 18:00:24.554946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:24.554931 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 18:00:24.555112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:24.555093 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:24.555154 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:24.555139 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:25.554988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:25.554954 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:25.555385 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:25.555009 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:26.555348 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:26.555308 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:26.555743 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:26.555367 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:27.555772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:27.555734 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:27.556145 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:27.555790 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:28.555238 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:28.555203 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:28.555412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:28.555270 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:29.555306 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:29.555268 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-62zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Sep 18 18:00:29.555676 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:29.555338 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" podUID="39ca5410-7955-4a3d-8890-f90cb83ce783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Sep 18 18:00:30.559152 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:30.559122 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 18:00:30.559506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:30.559398 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 18:00:30.560140 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:00:30.560122 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-62zx7" Sep 18 18:01:05.224358 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.224319 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz"] Sep 18 18:01:05.226477 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.226459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.229174 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.229150 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tq6dk\"" Sep 18 18:01:05.229308 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.229160 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 18:01:05.230042 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.230021 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 18:01:05.230121 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.230037 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 18 18:01:05.240556 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.240529 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz"] Sep 18 18:01:05.341783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341746 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.341783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.342018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.342018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxbt\" (UniqueName: \"kubernetes.io/projected/4180bffa-61bf-466d-aefc-d95a87cbfdca-kube-api-access-dmxbt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.342018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.342018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.341981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4180bffa-61bf-466d-aefc-d95a87cbfdca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443295 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4180bffa-61bf-466d-aefc-d95a87cbfdca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443385 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443375 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxbt\" (UniqueName: \"kubernetes.io/projected/4180bffa-61bf-466d-aefc-d95a87cbfdca-kube-api-access-dmxbt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443546 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443526 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443787 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443757 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.443787 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.443772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.445556 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.445535 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4180bffa-61bf-466d-aefc-d95a87cbfdca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.445702 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.445685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4180bffa-61bf-466d-aefc-d95a87cbfdca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.452621 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.452595 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxbt\" (UniqueName: \"kubernetes.io/projected/4180bffa-61bf-466d-aefc-d95a87cbfdca-kube-api-access-dmxbt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz\" (UID: \"4180bffa-61bf-466d-aefc-d95a87cbfdca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.536601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.536500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:05.817526 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.817502 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz"] Sep 18 18:01:05.820301 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:01:05.820277 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4180bffa_61bf_466d_aefc_d95a87cbfdca.slice/crio-8348b83d8e6162bb8f04a953ac07cc671ed669a612aee21718ccebcd39f9fbdc WatchSource:0}: Error finding container 8348b83d8e6162bb8f04a953ac07cc671ed669a612aee21718ccebcd39f9fbdc: Status 404 returned error can't find the container with id 8348b83d8e6162bb8f04a953ac07cc671ed669a612aee21718ccebcd39f9fbdc Sep 18 18:01:05.822066 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:05.822046 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 18:01:06.751096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:06.751057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" event={"ID":"4180bffa-61bf-466d-aefc-d95a87cbfdca","Type":"ContainerStarted","Data":"8348b83d8e6162bb8f04a953ac07cc671ed669a612aee21718ccebcd39f9fbdc"} Sep 18 18:01:07.757789 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:07.757748 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" event={"ID":"4180bffa-61bf-466d-aefc-d95a87cbfdca","Type":"ContainerStarted","Data":"d47b83d68d6909bc3141ddd08cd17965d1fcd091dc9f9d3f5bcbe7b34a72589c"} Sep 18 18:01:07.777432 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:07.777385 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" podStartSLOduration=1.836120138 podStartE2EDuration="2.777371816s" totalImagesPullingTime="941.251678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:07.775774616 +0000 UTC m=+672.868429740" watchObservedRunningTime="2026-09-18 18:01:07.777371816 +0000 UTC m=+672.870026924" Sep 18 18:01:15.536681 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:15.536639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:15.537089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:15.536708 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:15.548719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:15.548694 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:15.808023 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:15.807940 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz" Sep 18 18:01:23.016429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.016387 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m"] Sep 18 18:01:23.019474 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.019456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.022055 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.022033 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 18 18:01:23.033974 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.033954 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m"] Sep 18 18:01:23.086930 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.086894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0177179c-3cc8-4624-b7f2-66edec58b160-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.087057 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.086937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.087057 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.086962 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c577t\" (UniqueName: \"kubernetes.io/projected/0177179c-3cc8-4624-b7f2-66edec58b160-kube-api-access-c577t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.087131 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.087052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.087131 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.087081 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.087131 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.087101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188225 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0177179c-3cc8-4624-b7f2-66edec58b160-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c577t\" (UniqueName: \"kubernetes.io/projected/0177179c-3cc8-4624-b7f2-66edec58b160-kube-api-access-c577t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188558 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188558 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188694 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188753 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.188807 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.188753 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.190538 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.190518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0177179c-3cc8-4624-b7f2-66edec58b160-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.190643 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.190597 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0177179c-3cc8-4624-b7f2-66edec58b160-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.197296 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.197274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c577t\" (UniqueName: \"kubernetes.io/projected/0177179c-3cc8-4624-b7f2-66edec58b160-kube-api-access-c577t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m\" (UID: \"0177179c-3cc8-4624-b7f2-66edec58b160\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.329490 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.329404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:23.607578 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.607470 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m"] Sep 18 18:01:23.610226 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:01:23.610195 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0177179c_3cc8_4624_b7f2_66edec58b160.slice/crio-98216b89edb691dc185b30139a072e4b10f7d9fd47f6e2624aa01606eed715f2 WatchSource:0}: Error finding container 98216b89edb691dc185b30139a072e4b10f7d9fd47f6e2624aa01606eed715f2: Status 404 returned error can't find the container with id 98216b89edb691dc185b30139a072e4b10f7d9fd47f6e2624aa01606eed715f2 Sep 18 18:01:23.834595 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:23.834562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" event={"ID":"0177179c-3cc8-4624-b7f2-66edec58b160","Type":"ContainerStarted","Data":"98216b89edb691dc185b30139a072e4b10f7d9fd47f6e2624aa01606eed715f2"} Sep 18 18:01:24.841479 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:24.841439 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" event={"ID":"0177179c-3cc8-4624-b7f2-66edec58b160","Type":"ContainerStarted","Data":"e5786b48be88f649ff5f49c0c61378b350bf2c54b324420ccb9e38f010ed01a1"} Sep 18 18:01:33.329856 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:33.329815 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:33.330288 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:33.329873 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:33.342382 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:33.342356 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:01:33.375661 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:33.375594 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" podStartSLOduration=11.147933175 podStartE2EDuration="11.375573456s" totalImagesPullingTime="227.640281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:24.879543439 +0000 UTC m=+689.972198539" watchObservedRunningTime="2026-09-18 18:01:33.375573456 +0000 UTC m=+698.468228544" Sep 18 18:01:33.894242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:01:33.894215 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m" Sep 18 18:03:24.609989 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.609951 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:03:24.612103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.612087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.655292 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.655261 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-609aba-openshift-default-dockercfg-mhml4\"" Sep 18 18:03:24.681566 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681566 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.681801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681749 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.682022 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lslj\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.682022 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.681864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.700784 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.700752 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:03:24.783095 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lslj\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783255 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.783964 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783827 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.784051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.784051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.783876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.784242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.784219 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.785656 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.785633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.785888 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.785864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.816422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.816377 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:03:24.819034 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.819013 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.819292 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.819265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lslj\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.821481 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.821458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token\") pod \"e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.822246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.822231 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-03e512-openshift-default-dockercfg-2cw8c\"" Sep 18 18:03:24.847542 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.847512 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:03:24.884209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.884630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.884548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.922119 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.922087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985397 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.985735 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.985635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.986298 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.986164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.986976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.986470 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.986976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.986793 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.987603 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.987349 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.989310 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.989268 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.989751 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.989711 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:24.991588 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:24.991569 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:25.000627 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.000604 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:25.005093 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.005070 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms\") pod \"e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:25.069081 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.069047 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:03:25.071800 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.071780 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.075704 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.075683 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-502939-openshift-default-dockercfg-5gw2n\"" Sep 18 18:03:25.086018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.085995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4q4t\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086218 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086218 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086218 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086269 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.086328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.086301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.103423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.103397 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:03:25.141054 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.140978 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:25.187025 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.186989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187025 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4q4t\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.187291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.187679 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188545 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.188352 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188545 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.188396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188636 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.188588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.188772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.188700 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.190950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.190884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.191296 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.191271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.231970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.231938 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:03:25.233001 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:25.232975 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5773d6a9_a8a4_4d95_8112_6862993adac4.slice/crio-dfa67e263fbfd20db171d36bde707676f47b3ce9ac361750918e6f588d05d2b7 WatchSource:0}: Error finding container dfa67e263fbfd20db171d36bde707676f47b3ce9ac361750918e6f588d05d2b7: Status 404 returned error can't find the container with id dfa67e263fbfd20db171d36bde707676f47b3ce9ac361750918e6f588d05d2b7 Sep 18 18:03:25.233868 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.233752 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4q4t\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.235447 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.235418 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.235524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.235479 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.235524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.235509 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.235584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.235549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token\") pod \"e2e-inf-a-502939-openshift-default-545646f854-vnvkh\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.382175 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.382141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:25.404002 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.403962 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerStarted","Data":"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db"} Sep 18 18:03:25.404121 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.404020 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerStarted","Data":"dfa67e263fbfd20db171d36bde707676f47b3ce9ac361750918e6f588d05d2b7"} Sep 18 18:03:25.437371 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.434410 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:03:25.445602 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:25.445570 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf44764e_5a06_4b8f_b966_d8b3b69389f8.slice/crio-f9e02a00f0a02e6da90fd55e5a65558e698f0685f570464320ce33ea62b0450e WatchSource:0}: Error finding container f9e02a00f0a02e6da90fd55e5a65558e698f0685f570464320ce33ea62b0450e: Status 404 returned error can't find the container with id f9e02a00f0a02e6da90fd55e5a65558e698f0685f570464320ce33ea62b0450e Sep 18 18:03:25.448962 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.448927 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.449130 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.448998 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.449130 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.449038 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.451482 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.451180 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podStartSLOduration=1.451161821 podStartE2EDuration="1.451161821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:25.446552746 +0000 UTC m=+810.539207864" watchObservedRunningTime="2026-09-18 18:03:25.451161821 +0000 UTC m=+810.543816909" Sep 18 18:03:25.737779 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.737753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:03:25.740857 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:25.740826 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb5b098_d016_4c37_88b3_1d6864dff6f5.slice/crio-5c01fc1aa6f1fb66805cece93c57b44b4b90931ef5269fb604e6bf45a58a3104 WatchSource:0}: Error finding container 5c01fc1aa6f1fb66805cece93c57b44b4b90931ef5269fb604e6bf45a58a3104: Status 404 returned error can't find the container with id 5c01fc1aa6f1fb66805cece93c57b44b4b90931ef5269fb604e6bf45a58a3104 Sep 18 18:03:25.743905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.743870 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.744024 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.743942 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.744024 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.743976 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:25.922504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:25.922473 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:26.410705 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.410669 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerStarted","Data":"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70"} Sep 18 18:03:26.410871 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.410728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerStarted","Data":"f9e02a00f0a02e6da90fd55e5a65558e698f0685f570464320ce33ea62b0450e"} Sep 18 18:03:26.412322 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.412296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerStarted","Data":"1f0f37e8d8a306af148dc719b9cf79503ec988086430228f60632ec4b851a3a2"} Sep 18 18:03:26.412440 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.412336 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerStarted","Data":"5c01fc1aa6f1fb66805cece93c57b44b4b90931ef5269fb604e6bf45a58a3104"} Sep 18 18:03:26.456375 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.456322 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podStartSLOduration=2.456305231 podStartE2EDuration="2.456305231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:26.449382914 +0000 UTC m=+811.542037993" watchObservedRunningTime="2026-09-18 18:03:26.456305231 +0000 UTC m=+811.548960317" Sep 18 18:03:26.528374 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.528325 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podStartSLOduration=1.528309826 podStartE2EDuration="1.528309826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:26.525149891 +0000 UTC m=+811.617804990" watchObservedRunningTime="2026-09-18 18:03:26.528309826 +0000 UTC m=+811.620964912" Sep 18 18:03:26.927826 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:26.927790 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:27.141141 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.141112 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:27.145919 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.145893 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:27.382523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.382437 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:27.387084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.387061 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:27.418571 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.418489 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:27.418571 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.418538 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:27.418571 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.418564 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:27.419127 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.419106 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:27.419417 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.419390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:27.419480 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:27.419444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:36.155896 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.155861 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:03:36.164110 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.164090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.179242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.179216 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:03:36.188054 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.188033 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-609aba\"" Sep 18 18:03:36.188906 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.188883 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-609aba-dockercfg-wc2tf\"" Sep 18 18:03:36.285571 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.285540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w28g\" (UniqueName: \"kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.285719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.285627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.386047 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.386017 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.386224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.386075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w28g\" (UniqueName: \"kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.386732 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.386711 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.421581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.421511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w28g\" (UniqueName: \"kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g\") pod \"payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.474734 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.474706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:36.806327 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:36.806033 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:03:36.807505 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:36.807467 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bd9803_e7b5_4d21_8660_d0c1000e555a.slice/crio-d468f53e6072ab01b979b237ee4e4289fa0c70bf328a71faf6cd040041a5dc51 WatchSource:0}: Error finding container d468f53e6072ab01b979b237ee4e4289fa0c70bf328a71faf6cd040041a5dc51: Status 404 returned error can't find the container with id d468f53e6072ab01b979b237ee4e4289fa0c70bf328a71faf6cd040041a5dc51 Sep 18 18:03:37.468683 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:37.468649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" event={"ID":"d9bd9803-e7b5-4d21-8660-d0c1000e555a","Type":"ContainerStarted","Data":"d468f53e6072ab01b979b237ee4e4289fa0c70bf328a71faf6cd040041a5dc51"} Sep 18 18:03:39.500899 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.500861 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" event={"ID":"d9bd9803-e7b5-4d21-8660-d0c1000e555a","Type":"ContainerStarted","Data":"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312"} Sep 18 18:03:39.501503 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.501479 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:39.530221 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.530149 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" podStartSLOduration=1.3887903449999999 podStartE2EDuration="3.530127805s" totalImagesPullingTime="2.14133746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:39.527454184 +0000 UTC m=+824.620109288" watchObservedRunningTime="2026-09-18 18:03:39.530127805 +0000 UTC m=+824.622782893" Sep 18 18:03:39.644521 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.640136 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:03:39.648145 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.648124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.653750 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.653486 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-63da2c01-openshift-default-dockercfg-kg56l\"" Sep 18 18:03:39.677483 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.677455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:03:39.741279 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.741237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.741543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.741522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.741662 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.741645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnggf\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.741783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.741768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.741962 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.741946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.742084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.742070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.742226 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.742210 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.742355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.742340 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.742475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.742459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.843916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.843978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnggf\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.844630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.846206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.844734 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.846206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.845144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.846206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.845427 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.846206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.845696 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.848064 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.848037 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.849059 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.849038 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.856030 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.855931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.856896 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.856871 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnggf\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf\") pod \"e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:39.963505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:39.963464 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:40.326449 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.326398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:03:40.335547 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:40.335482 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618b6682_d0e8_41a0_9721_c0aded0bbddb.slice/crio-44e296ca8a25ae3412e76cd0cf3875d922de4f272003fe6e88c49c59dfb301ea WatchSource:0}: Error finding container 44e296ca8a25ae3412e76cd0cf3875d922de4f272003fe6e88c49c59dfb301ea: Status 404 returned error can't find the container with id 44e296ca8a25ae3412e76cd0cf3875d922de4f272003fe6e88c49c59dfb301ea Sep 18 18:03:40.339217 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.339001 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:40.339217 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.339093 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:40.339217 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.339138 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:40.515571 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.515532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerStarted","Data":"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235"} Sep 18 18:03:40.516036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.515591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerStarted","Data":"44e296ca8a25ae3412e76cd0cf3875d922de4f272003fe6e88c49c59dfb301ea"} Sep 18 18:03:40.556917 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.556864 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podStartSLOduration=1.556846203 podStartE2EDuration="1.556846203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:40.543967901 +0000 UTC m=+825.636623018" watchObservedRunningTime="2026-09-18 18:03:40.556846203 +0000 UTC m=+825.649501330" Sep 18 18:03:40.963884 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:40.963855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:41.969293 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:41.969260 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:42.530103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:42.530065 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:42.534444 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:42.534410 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:03:44.243840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.243798 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:03:44.247036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.247009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.255743 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.255452 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ee3049-openshift-default-dockercfg-62qw4\"" Sep 18 18:03:44.327336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.327236 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:03:44.390887 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.390848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391060 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.390896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391060 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.390986 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbwp\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391100 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391317 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391317 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.391414 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.391324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492134 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492510 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbwp\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492510 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492510 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492510 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492766 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.492972 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.492818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.493095 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.493062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.493529 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.493448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.493529 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.493486 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.493659 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.493601 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.495365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.495289 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.495776 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.495751 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.503526 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.503447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbwp\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.506299 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.506276 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token\") pod \"e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.559275 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.559241 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:44.857742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.857699 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:03:44.859987 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:03:44.859962 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fbced2_da3a_48d1_89ca_f02b186d576a.slice/crio-dba4510f98ccf80b0ac8a38cfcce1c91c06e8b52597f2847029142189662804e WatchSource:0}: Error finding container dba4510f98ccf80b0ac8a38cfcce1c91c06e8b52597f2847029142189662804e: Status 404 returned error can't find the container with id dba4510f98ccf80b0ac8a38cfcce1c91c06e8b52597f2847029142189662804e Sep 18 18:03:44.862628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.862590 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:44.862727 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.862707 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:44.862798 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:44.862781 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:45.550938 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:45.550884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerStarted","Data":"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd"} Sep 18 18:03:45.551521 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:45.551499 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerStarted","Data":"dba4510f98ccf80b0ac8a38cfcce1c91c06e8b52597f2847029142189662804e"} Sep 18 18:03:45.560213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:45.560177 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:45.565443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:45.565413 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:45.593913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:45.593849 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podStartSLOduration=1.593829585 podStartE2EDuration="1.593829585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:45.592018807 +0000 UTC m=+830.684673912" watchObservedRunningTime="2026-09-18 18:03:45.593829585 +0000 UTC m=+830.686484673" Sep 18 18:03:46.558004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.557923 2570 generic.go:356] "Generic (PLEG): container finished" podID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerID="1f0f37e8d8a306af148dc719b9cf79503ec988086430228f60632ec4b851a3a2" exitCode=0 Sep 18 18:03:46.558004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.557988 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerDied","Data":"1f0f37e8d8a306af148dc719b9cf79503ec988086430228f60632ec4b851a3a2"} Sep 18 18:03:46.558541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.558337 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:46.558541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.558383 2570 scope.go:117] "RemoveContainer" containerID="1f0f37e8d8a306af148dc719b9cf79503ec988086430228f60632ec4b851a3a2" Sep 18 18:03:46.559213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.559179 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:03:46.559537 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.559514 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:46.559593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.559581 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:46.559629 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:46.559614 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:47.565350 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.565296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerStarted","Data":"62340a5c352b3c898dbcf6e059c3744f134163e3877bdb51e9f19e638aeee0f2"} Sep 18 18:03:47.567105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.567082 2570 generic.go:356] "Generic (PLEG): container finished" podID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerID="8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70" exitCode=0 Sep 18 18:03:47.567228 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.567164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerDied","Data":"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70"} Sep 18 18:03:47.567638 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.567623 2570 scope.go:117] "RemoveContainer" containerID="8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70" Sep 18 18:03:47.574016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.569676 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:47.574016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.569856 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:47.574016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:47.569963 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:48.382375 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:48.382334 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:48.383064 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:48.382946 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 18 18:03:48.383260 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:48.383236 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 18 18:03:48.574691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:48.574653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerStarted","Data":"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f"} Sep 18 18:03:49.142141 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:49.142097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:49.142531 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:49.142500 2570 patch_prober.go:28] interesting pod/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 18 18:03:49.142642 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:49.142560 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 18 18:03:49.383242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:49.383203 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 18 18:03:49.383403 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:49.383269 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 18 18:03:50.141674 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:50.141627 2570 patch_prober.go:28] interesting pod/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 18 18:03:50.142210 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:50.141699 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 18 18:03:50.383077 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:50.383046 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 18 18:03:50.383325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:50.383300 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 18 18:03:50.518406 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:50.518361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:03:51.142076 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:51.142035 2570 patch_prober.go:28] interesting pod/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 18 18:03:51.142588 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:51.142100 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 18 18:03:51.382617 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:51.382562 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 18 18:03:51.382814 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:51.382632 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 18 18:03:52.141577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:52.141493 2570 patch_prober.go:28] interesting pod/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 18 18:03:52.141577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:52.141548 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 18 18:03:52.382895 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:52.382853 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 18 18:03:52.383250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:52.382920 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 18 18:03:53.142176 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:53.142114 2570 patch_prober.go:28] interesting pod/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 18 18:03:53.142370 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:53.142231 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 18 18:03:53.574040 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:53.572562 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:53.574040 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:53.573875 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:53.574955 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:53.574888 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:03:54.242266 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:54.242220 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:54.242815 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:54.242788 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:54.244466 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:54.244441 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:03:55.760803 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.760757 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:03:55.763004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.762986 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:55.766997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.766973 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ee3049-dockercfg-wvjqq\"" Sep 18 18:03:55.767311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.767283 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ee3049\"" Sep 18 18:03:55.781739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.781716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:03:55.943498 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.943406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtzf\" (UniqueName: \"kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:55.943498 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:55.943465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.044780 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.044745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtzf\" (UniqueName: \"kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.044940 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.044803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.045553 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.045534 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.059462 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.059433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtzf\" (UniqueName: \"kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf\") pod \"payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.073405 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.073378 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.394486 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.385533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:03:56.647583 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.647495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" event={"ID":"1a9cfb68-50b7-493a-af16-7f2e3c4aa254","Type":"ContainerStarted","Data":"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee"} Sep 18 18:03:56.647583 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.647549 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" event={"ID":"1a9cfb68-50b7-493a-af16-7f2e3c4aa254","Type":"ContainerStarted","Data":"279ffa64ad53dd9403c7c8717fc32be16708e77d704b7c4185e0cf81a3642d16"} Sep 18 18:03:56.647777 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.647590 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:03:56.649270 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.649249 2570 generic.go:356] "Generic (PLEG): container finished" podID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerID="903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db" exitCode=0 Sep 18 18:03:56.649398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.649274 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerDied","Data":"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db"} Sep 18 18:03:56.649588 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.649576 2570 scope.go:117] "RemoveContainer" containerID="903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db" Sep 18 18:03:56.650577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.650549 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:56.650672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.650613 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:56.650672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.650642 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:03:56.674441 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:56.674399 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" podStartSLOduration=1.674386761 podStartE2EDuration="1.674386761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:56.67269393 +0000 UTC m=+841.765349018" watchObservedRunningTime="2026-09-18 18:03:56.674386761 +0000 UTC m=+841.767041848" Sep 18 18:03:57.421375 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:57.421274 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:57.657768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:57.657727 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerStarted","Data":"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe"} Sep 18 18:03:57.925210 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:57.924417 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:03:57.925210 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:57.924542 2570 patch_prober.go:28] interesting pod/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 18 18:03:57.925210 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:57.924585 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 18 18:03:58.923387 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:58.923335 2570 patch_prober.go:28] interesting pod/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 18 18:03:58.923881 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:58.923423 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 18 18:03:59.922905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:59.922871 2570 patch_prober.go:28] interesting pod/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 18 18:03:59.923059 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:03:59.922929 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 18 18:04:00.922588 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:00.922546 2570 patch_prober.go:28] interesting pod/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 18 18:04:00.922971 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:00.922611 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 18 18:04:01.922634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:01.922583 2570 patch_prober.go:28] interesting pod/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 18 18:04:01.923113 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:01.922657 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 18 18:04:03.036089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.036056 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:04:03.036934 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.036906 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:04:03.037774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.037756 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:04:03.297263 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.287944 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf"] Sep 18 18:04:03.303238 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.303030 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:03.310579 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.310552 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:04:03.311252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.310780 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:04:03.313953 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.313920 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf"] Sep 18 18:04:03.322454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.322417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf\" (UID: \"9d011056-63f8-4b45-b1d3-6e7c7d9739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:03.423923 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.423887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf\" (UID: \"9d011056-63f8-4b45-b1d3-6e7c7d9739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:03.424789 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.424760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf\" (UID: \"9d011056-63f8-4b45-b1d3-6e7c7d9739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:03.628537 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.628500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:03.718853 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.717963 2570 generic.go:356] "Generic (PLEG): container finished" podID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerID="0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235" exitCode=0 Sep 18 18:04:03.718853 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.718560 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerDied","Data":"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235"} Sep 18 18:04:03.719095 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.718938 2570 scope.go:117] "RemoveContainer" containerID="0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235" Sep 18 18:04:03.720442 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.720382 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:03.720508 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.720472 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:03.720547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.720520 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:03.925384 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:03.925351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf"] Sep 18 18:04:03.927603 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:04:03.927557 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d011056_63f8_4b45_b1d3_6e7c7d9739bd.slice/crio-19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626 WatchSource:0}: Error finding container 19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626: Status 404 returned error can't find the container with id 19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626 Sep 18 18:04:04.735058 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:04.735019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerStarted","Data":"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039"} Sep 18 18:04:04.738596 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:04.738561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" event={"ID":"9d011056-63f8-4b45-b1d3-6e7c7d9739bd","Type":"ContainerStarted","Data":"19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626"} Sep 18 18:04:04.964261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:04.964233 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:04.964443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:04.964323 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:04.964443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:04.964361 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:05.963976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:05.963937 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:05.964363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:05.964014 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:06.964429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:06.964392 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:06.964840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:06.964454 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:07.660146 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:07.660116 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:04:07.964768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:07.964686 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:07.964768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:07.964736 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:08.772093 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:08.772055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" event={"ID":"9d011056-63f8-4b45-b1d3-6e7c7d9739bd","Type":"ContainerStarted","Data":"493d2ad89f6b14a0f86043d0d6f773cb4b5fad62a183ba26c9ec1b6c41245b77"} Sep 18 18:04:08.798170 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:08.798109 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" podStartSLOduration=1.977751112 podStartE2EDuration="5.798084556s" totalImagesPullingTime="3.820333444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:08.795518546 +0000 UTC m=+853.888173657" watchObservedRunningTime="2026-09-18 18:04:08.798084556 +0000 UTC m=+853.890739647" Sep 18 18:04:08.964069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:08.964035 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:08.964265 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:08.964090 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:09.779456 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:09.779423 2570 generic.go:356] "Generic (PLEG): container finished" podID="9d011056-63f8-4b45-b1d3-6e7c7d9739bd" containerID="493d2ad89f6b14a0f86043d0d6f773cb4b5fad62a183ba26c9ec1b6c41245b77" exitCode=0 Sep 18 18:04:09.779842 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:09.779507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" event={"ID":"9d011056-63f8-4b45-b1d3-6e7c7d9739bd","Type":"ContainerDied","Data":"493d2ad89f6b14a0f86043d0d6f773cb4b5fad62a183ba26c9ec1b6c41245b77"} Sep 18 18:04:09.964026 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:09.963981 2570 patch_prober.go:28] interesting pod/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 18 18:04:09.964239 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:09.964043 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 18 18:04:10.921920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:10.921888 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:10.969143 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:10.969110 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:10.969547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:10.969487 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:10.970368 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:10.970349 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:11.012420 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.012392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca\") pod \"9d011056-63f8-4b45-b1d3-6e7c7d9739bd\" (UID: \"9d011056-63f8-4b45-b1d3-6e7c7d9739bd\") " Sep 18 18:04:11.012786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.012765 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d011056-63f8-4b45-b1d3-6e7c7d9739bd" (UID: "9d011056-63f8-4b45-b1d3-6e7c7d9739bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:04:11.114051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.113956 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d011056-63f8-4b45-b1d3-6e7c7d9739bd-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:11.792018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.791986 2570 generic.go:356] "Generic (PLEG): container finished" podID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerID="eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd" exitCode=0 Sep 18 18:04:11.792268 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.792053 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerDied","Data":"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd"} Sep 18 18:04:11.792513 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.792493 2570 scope.go:117] "RemoveContainer" containerID="eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd" Sep 18 18:04:11.793708 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793681 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" event={"ID":"9d011056-63f8-4b45-b1d3-6e7c7d9739bd","Type":"ContainerDied","Data":"19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626"} Sep 18 18:04:11.793775 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793738 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19302ede55cec847ec420abe74aa5f743da860a35e972c76c54fa5ddb0f5b626" Sep 18 18:04:11.793775 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793699 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf" Sep 18 18:04:11.793844 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793787 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:11.793880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793857 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:11.793913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:11.793888 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:04:12.800193 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:12.800153 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerStarted","Data":"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a"} Sep 18 18:04:13.559578 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:13.559544 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:04:13.560594 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:13.560566 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 18 18:04:13.560724 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:13.560612 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 18 18:04:14.379666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.379623 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf"] Sep 18 18:04:14.403209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.402730 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63da2c01-7fjkf"] Sep 18 18:04:14.547628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.547593 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:04:14.547801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.547780 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" containerID="cri-o://732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039" gracePeriod=30 Sep 18 18:04:14.560249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.560219 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 18 18:04:14.560386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.560277 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 18 18:04:14.941868 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:14.941832 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d011056-63f8-4b45-b1d3-6e7c7d9739bd" path="/var/lib/kubelet/pods/9d011056-63f8-4b45-b1d3-6e7c7d9739bd/volumes" Sep 18 18:04:15.560500 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:15.560460 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 18 18:04:15.560905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:15.560518 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 18 18:04:16.558243 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:16.558208 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:04:16.559830 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:16.559810 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 18 18:04:16.559883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:16.559857 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 18 18:04:17.559863 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:17.559825 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 18 18:04:17.560285 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:17.559894 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 18 18:04:18.619608 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:18.619551 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:04:18.622880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:18.622843 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:04:19.797170 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.797147 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:19.846606 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.846533 2570 generic.go:356] "Generic (PLEG): container finished" podID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerID="732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039" exitCode=0 Sep 18 18:04:19.846606 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.846579 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" Sep 18 18:04:19.846813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.846581 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerDied","Data":"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039"} Sep 18 18:04:19.846813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.846708 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d" event={"ID":"618b6682-d0e8-41a0-9721-c0aded0bbddb","Type":"ContainerDied","Data":"44e296ca8a25ae3412e76cd0cf3875d922de4f272003fe6e88c49c59dfb301ea"} Sep 18 18:04:19.846813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.846741 2570 scope.go:117] "RemoveContainer" containerID="732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039" Sep 18 18:04:19.855850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.855832 2570 scope.go:117] "RemoveContainer" containerID="0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235" Sep 18 18:04:19.864115 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.864098 2570 scope.go:117] "RemoveContainer" containerID="732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039" Sep 18 18:04:19.864386 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:04:19.864366 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039\": container with ID starting with 732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039 not found: ID does not exist" containerID="732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039" Sep 18 18:04:19.864470 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.864396 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039"} err="failed to get container status \"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039\": rpc error: code = NotFound desc = could not find container \"732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039\": container with ID starting with 732c3e38074420f1232d3864a8ca0ac6b5bf8fb7741de521849aa79826bbc039 not found: ID does not exist" Sep 18 18:04:19.864470 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.864415 2570 scope.go:117] "RemoveContainer" containerID="0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235" Sep 18 18:04:19.864679 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:04:19.864660 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235\": container with ID starting with 0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235 not found: ID does not exist" containerID="0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235" Sep 18 18:04:19.864802 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.864685 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235"} err="failed to get container status \"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235\": rpc error: code = NotFound desc = could not find container \"0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235\": container with ID starting with 0d8bb4935f071f2dc31537d3184a737318a3183b1fecbef0c9c9c0a175f78235 not found: ID does not exist" Sep 18 18:04:19.903807 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903783 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.903901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903817 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.903901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903843 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.903901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903865 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904045 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904045 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.903971 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904045 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904013 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904058 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnggf\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904047 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:04:19.904213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904101 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket\") pod \"618b6682-d0e8-41a0-9721-c0aded0bbddb\" (UID: \"618b6682-d0e8-41a0-9721-c0aded0bbddb\") " Sep 18 18:04:19.904390 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904273 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:04:19.904390 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904307 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data" (OuterVolumeSpecName: "istio-data") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:04:19.904390 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904329 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:04:19.904543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904418 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:19.904543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904435 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:19.904543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904444 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b6682-d0e8-41a0-9721-c0aded0bbddb-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:19.904543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904452 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:19.904543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.904457 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:04:19.906131 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.906025 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token" (OuterVolumeSpecName: "istio-token") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:04:19.906131 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.906098 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:04:19.906261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.906145 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf" (OuterVolumeSpecName: "kube-api-access-dnggf") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "kube-api-access-dnggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:04:19.906261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:19.906205 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "618b6682-d0e8-41a0-9721-c0aded0bbddb" (UID: "618b6682-d0e8-41a0-9721-c0aded0bbddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:04:20.005768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.005733 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:20.005768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.005764 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnggf\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-kube-api-access-dnggf\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:20.005768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.005775 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b6682-d0e8-41a0-9721-c0aded0bbddb-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:20.005993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.005785 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:20.005993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.005794 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b6682-d0e8-41a0-9721-c0aded0bbddb-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:20.183839 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.183809 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:04:20.190836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.190810 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-63da2c01-openshift-default-666d4bbccb-4ff6d"] Sep 18 18:04:20.942596 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:20.942549 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" path="/var/lib/kubelet/pods/618b6682-d0e8-41a0-9721-c0aded0bbddb/volumes" Sep 18 18:04:24.941870 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.941837 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942134 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d011056-63f8-4b45-b1d3-6e7c7d9739bd" containerName="revoke-keys" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942144 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d011056-63f8-4b45-b1d3-6e7c7d9739bd" containerName="revoke-keys" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942167 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942172 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942179 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942198 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942247 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942255 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="618b6682-d0e8-41a0-9721-c0aded0bbddb" containerName="istio-proxy" Sep 18 18:04:24.942602 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.942261 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d011056-63f8-4b45-b1d3-6e7c7d9739bd" containerName="revoke-keys" Sep 18 18:04:24.945407 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.945388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:24.948586 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.948567 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9f929078-dockercfg-xtvct\"" Sep 18 18:04:24.948865 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.948851 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9f929078\"" Sep 18 18:04:24.965797 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.965775 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:04:24.993968 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.993924 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:04:24.996826 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:24.996808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.029232 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.029171 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:04:25.048207 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.047877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8299t\" (UniqueName: \"kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.048207 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.047975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.048207 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.048048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.048207 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.048074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5tk\" (UniqueName: \"kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.149471 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.149433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.149657 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.149512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.149657 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.149536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5tk\" (UniqueName: \"kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.149657 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.149614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8299t\" (UniqueName: \"kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.150202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.150162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.150314 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.150260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.171989 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.171958 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8299t\" (UniqueName: \"kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t\") pod \"payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.172118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.171966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5tk\" (UniqueName: \"kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk\") pod \"payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.255648 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.255559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:25.306515 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.306489 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.533029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.532993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:04:25.606615 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:04:25.606579 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5405e92_47bf_4a6c_868f_378fd8376557.slice/crio-a0384285601019431ab12dc2d59a4c9043844a1f5905735a5c9d8282460448ad WatchSource:0}: Error finding container a0384285601019431ab12dc2d59a4c9043844a1f5905735a5c9d8282460448ad: Status 404 returned error can't find the container with id a0384285601019431ab12dc2d59a4c9043844a1f5905735a5c9d8282460448ad Sep 18 18:04:25.607840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.607820 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:04:25.885480 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.885386 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" event={"ID":"f5405e92-47bf-4a6c-868f-378fd8376557","Type":"ContainerStarted","Data":"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748"} Sep 18 18:04:25.885480 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.885448 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" event={"ID":"f5405e92-47bf-4a6c-868f-378fd8376557","Type":"ContainerStarted","Data":"a0384285601019431ab12dc2d59a4c9043844a1f5905735a5c9d8282460448ad"} Sep 18 18:04:25.885684 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.885560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:25.887104 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.887078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" event={"ID":"4a9eb368-1450-4221-949e-7707708c92ae","Type":"ContainerStarted","Data":"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062"} Sep 18 18:04:25.887264 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.887117 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" event={"ID":"4a9eb368-1450-4221-949e-7707708c92ae","Type":"ContainerStarted","Data":"c51df98693ad3055edfaf09949991ed279a4160ce22272b781ebb11a446ef29a"} Sep 18 18:04:25.887264 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:25.887137 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:26.003933 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:26.003868 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" podStartSLOduration=2.00384681 podStartE2EDuration="2.00384681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:25.932121098 +0000 UTC m=+871.024776204" watchObservedRunningTime="2026-09-18 18:04:26.00384681 +0000 UTC m=+871.096501898" Sep 18 18:04:26.004557 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:26.004524 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" podStartSLOduration=2.004513598 podStartE2EDuration="2.004513598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:26.003274978 +0000 UTC m=+871.095930093" watchObservedRunningTime="2026-09-18 18:04:26.004513598 +0000 UTC m=+871.097168685" Sep 18 18:04:36.893952 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:36.893920 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:04:36.894503 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:36.894003 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:04:48.103008 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.100302 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr"] Sep 18 18:04:48.104287 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.104254 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:48.107780 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.107756 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:04:48.108010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.107992 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:04:48.112843 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.112819 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr"] Sep 18 18:04:48.267229 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.267177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr\" (UID: \"79682b52-7499-4486-8503-611ecb28b44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:48.368352 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.368249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr\" (UID: \"79682b52-7499-4486-8503-611ecb28b44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:48.368966 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.368937 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr\" (UID: \"79682b52-7499-4486-8503-611ecb28b44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:48.434349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.434312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:48.716709 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:48.716682 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr"] Sep 18 18:04:48.719035 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:04:48.719009 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79682b52_7499_4486_8503_611ecb28b44c.slice/crio-0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127 WatchSource:0}: Error finding container 0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127: Status 404 returned error can't find the container with id 0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127 Sep 18 18:04:49.016009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:49.015916 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" event={"ID":"79682b52-7499-4486-8503-611ecb28b44c","Type":"ContainerStarted","Data":"22183f632f4a6917f9d0fc258b13cddcc67330d00e0196ad4406e25ab55f81f4"} Sep 18 18:04:49.016009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:49.015977 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" event={"ID":"79682b52-7499-4486-8503-611ecb28b44c","Type":"ContainerStarted","Data":"0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127"} Sep 18 18:04:49.053469 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:49.053418 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" podStartSLOduration=1.053399813 podStartE2EDuration="1.053399813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:49.0492815 +0000 UTC m=+894.141936579" watchObservedRunningTime="2026-09-18 18:04:49.053399813 +0000 UTC m=+894.146054891" Sep 18 18:04:50.022620 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:50.022590 2570 generic.go:356] "Generic (PLEG): container finished" podID="79682b52-7499-4486-8503-611ecb28b44c" containerID="22183f632f4a6917f9d0fc258b13cddcc67330d00e0196ad4406e25ab55f81f4" exitCode=0 Sep 18 18:04:50.023017 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:50.022682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" event={"ID":"79682b52-7499-4486-8503-611ecb28b44c","Type":"ContainerDied","Data":"22183f632f4a6917f9d0fc258b13cddcc67330d00e0196ad4406e25ab55f81f4"} Sep 18 18:04:51.156397 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:51.154800 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:51.296457 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:51.296365 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca\") pod \"79682b52-7499-4486-8503-611ecb28b44c\" (UID: \"79682b52-7499-4486-8503-611ecb28b44c\") " Sep 18 18:04:51.296716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:51.296693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79682b52-7499-4486-8503-611ecb28b44c" (UID: "79682b52-7499-4486-8503-611ecb28b44c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:04:51.398027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:51.397984 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79682b52-7499-4486-8503-611ecb28b44c-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:04:52.039505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:52.039476 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" Sep 18 18:04:52.039710 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:52.039512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr" event={"ID":"79682b52-7499-4486-8503-611ecb28b44c","Type":"ContainerDied","Data":"0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127"} Sep 18 18:04:52.039710 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:04:52.039581 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0024762d633b34f12960ce489a00f12cf24b1927050b952306ce15208fa8e127" Sep 18 18:05:00.185324 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.185291 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:05:00.185791 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.185512 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" podUID="f5405e92-47bf-4a6c-868f-378fd8376557" containerName="payload-processing" containerID="cri-o://30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748" gracePeriod=30 Sep 18 18:05:00.217422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.217386 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:05:00.217587 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.217568 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" podUID="4a9eb368-1450-4221-949e-7707708c92ae" containerName="payload-pre-processing" containerID="cri-o://0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062" gracePeriod=30 Sep 18 18:05:00.513490 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.513464 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:05:00.516491 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.516467 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:05:00.526420 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.526392 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr"] Sep 18 18:05:00.529564 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.529543 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9f929078-p6xwr"] Sep 18 18:05:00.576000 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.575963 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume\") pod \"f5405e92-47bf-4a6c-868f-378fd8376557\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " Sep 18 18:05:00.576419 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.576392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr5tk\" (UniqueName: \"kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk\") pod \"4a9eb368-1450-4221-949e-7707708c92ae\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " Sep 18 18:05:00.576888 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.576852 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8299t\" (UniqueName: \"kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t\") pod \"f5405e92-47bf-4a6c-868f-378fd8376557\" (UID: \"f5405e92-47bf-4a6c-868f-378fd8376557\") " Sep 18 18:05:00.576996 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.576917 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume\") pod \"4a9eb368-1450-4221-949e-7707708c92ae\" (UID: \"4a9eb368-1450-4221-949e-7707708c92ae\") " Sep 18 18:05:00.576996 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.576860 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5405e92-47bf-4a6c-868f-378fd8376557" (UID: "f5405e92-47bf-4a6c-868f-378fd8376557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:00.577395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.577361 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a9eb368-1450-4221-949e-7707708c92ae" (UID: "4a9eb368-1450-4221-949e-7707708c92ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:00.577691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.577666 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9eb368-1450-4221-949e-7707708c92ae-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:00.577691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.577690 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5405e92-47bf-4a6c-868f-378fd8376557-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:00.579943 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.579879 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk" (OuterVolumeSpecName: "kube-api-access-hr5tk") pod "4a9eb368-1450-4221-949e-7707708c92ae" (UID: "4a9eb368-1450-4221-949e-7707708c92ae"). InnerVolumeSpecName "kube-api-access-hr5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:00.580371 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.580325 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t" (OuterVolumeSpecName: "kube-api-access-8299t") pod "f5405e92-47bf-4a6c-868f-378fd8376557" (UID: "f5405e92-47bf-4a6c-868f-378fd8376557"). InnerVolumeSpecName "kube-api-access-8299t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:00.678838 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.678801 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5tk\" (UniqueName: \"kubernetes.io/projected/4a9eb368-1450-4221-949e-7707708c92ae-kube-api-access-hr5tk\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:00.678838 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.678832 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8299t\" (UniqueName: \"kubernetes.io/projected/f5405e92-47bf-4a6c-868f-378fd8376557-kube-api-access-8299t\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:00.941278 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:00.941248 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79682b52-7499-4486-8503-611ecb28b44c" path="/var/lib/kubelet/pods/79682b52-7499-4486-8503-611ecb28b44c/volumes" Sep 18 18:05:01.089017 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.088978 2570 generic.go:356] "Generic (PLEG): container finished" podID="f5405e92-47bf-4a6c-868f-378fd8376557" containerID="30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748" exitCode=0 Sep 18 18:05:01.089222 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.089034 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" Sep 18 18:05:01.089222 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.089028 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" event={"ID":"f5405e92-47bf-4a6c-868f-378fd8376557","Type":"ContainerDied","Data":"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748"} Sep 18 18:05:01.089339 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.089224 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll" event={"ID":"f5405e92-47bf-4a6c-868f-378fd8376557","Type":"ContainerDied","Data":"a0384285601019431ab12dc2d59a4c9043844a1f5905735a5c9d8282460448ad"} Sep 18 18:05:01.089339 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.089260 2570 scope.go:117] "RemoveContainer" containerID="30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748" Sep 18 18:05:01.090618 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.090593 2570 generic.go:356] "Generic (PLEG): container finished" podID="4a9eb368-1450-4221-949e-7707708c92ae" containerID="0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062" exitCode=0 Sep 18 18:05:01.090733 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.090667 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" Sep 18 18:05:01.090733 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.090686 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" event={"ID":"4a9eb368-1450-4221-949e-7707708c92ae","Type":"ContainerDied","Data":"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062"} Sep 18 18:05:01.090845 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.090730 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg" event={"ID":"4a9eb368-1450-4221-949e-7707708c92ae","Type":"ContainerDied","Data":"c51df98693ad3055edfaf09949991ed279a4160ce22272b781ebb11a446ef29a"} Sep 18 18:05:01.098483 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.098464 2570 scope.go:117] "RemoveContainer" containerID="30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748" Sep 18 18:05:01.098740 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:01.098719 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748\": container with ID starting with 30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748 not found: ID does not exist" containerID="30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748" Sep 18 18:05:01.098793 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.098751 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748"} err="failed to get container status \"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748\": rpc error: code = NotFound desc = could not find container \"30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748\": container with ID starting with 30e4764dfb3235cb20bfbfde32164125f9abe0d5996be880d230f72c8c95d748 not found: ID does not exist" Sep 18 18:05:01.098793 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.098768 2570 scope.go:117] "RemoveContainer" containerID="0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062" Sep 18 18:05:01.106701 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.106675 2570 scope.go:117] "RemoveContainer" containerID="0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062" Sep 18 18:05:01.106955 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:01.106934 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062\": container with ID starting with 0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062 not found: ID does not exist" containerID="0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062" Sep 18 18:05:01.107007 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.106962 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062"} err="failed to get container status \"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062\": rpc error: code = NotFound desc = could not find container \"0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062\": container with ID starting with 0409eeb3d5bccacd3bb7500b937c6c67870b1252c70f9ae81dbdf3000899a062 not found: ID does not exist" Sep 18 18:05:01.112982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.112957 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:05:01.116912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.116890 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9f929078-fbbb5d5f8-24vhg"] Sep 18 18:05:01.139898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.139856 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:05:01.151021 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.150980 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9f929078-5b6bddf894-6jlll"] Sep 18 18:05:01.598834 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.598802 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:01.599322 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599303 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79682b52-7499-4486-8503-611ecb28b44c" containerName="revoke-keys" Sep 18 18:05:01.599399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599325 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="79682b52-7499-4486-8503-611ecb28b44c" containerName="revoke-keys" Sep 18 18:05:01.599399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599349 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9eb368-1450-4221-949e-7707708c92ae" containerName="payload-pre-processing" Sep 18 18:05:01.599399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599358 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9eb368-1450-4221-949e-7707708c92ae" containerName="payload-pre-processing" Sep 18 18:05:01.599399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599373 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5405e92-47bf-4a6c-868f-378fd8376557" containerName="payload-processing" Sep 18 18:05:01.599399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599382 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5405e92-47bf-4a6c-868f-378fd8376557" containerName="payload-processing" Sep 18 18:05:01.599640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599480 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9eb368-1450-4221-949e-7707708c92ae" containerName="payload-pre-processing" Sep 18 18:05:01.599640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599492 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="79682b52-7499-4486-8503-611ecb28b44c" containerName="revoke-keys" Sep 18 18:05:01.599640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.599506 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5405e92-47bf-4a6c-868f-378fd8376557" containerName="payload-processing" Sep 18 18:05:01.602133 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.602110 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.609095 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.609075 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-284e39c5-openshift-default-dockercfg-2mv9r\"" Sep 18 18:05:01.622021 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.621996 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:01.688539 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.688761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.688761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.688761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.688761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.688761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcwd\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.689084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688766 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.689084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.689084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.688830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.789927 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.789883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.789937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.789968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.789998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790457 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790457 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790457 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcwd\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790457 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790440 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790616 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790470 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790616 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790552 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790742 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.790934 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.790912 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.792505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.792478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.792634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.792614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.800260 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.800234 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.802278 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.802255 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcwd\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd\") pod \"e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:01.912400 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:01.912369 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:02.209638 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.209608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:02.211134 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:02.211107 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7403d04a_361e_4e6c_bc0f_298efe916f38.slice/crio-21800e133bfe1f6dfcac0efb384a21e202a2aa8e4a7cb6ff6370382a0b968bea WatchSource:0}: Error finding container 21800e133bfe1f6dfcac0efb384a21e202a2aa8e4a7cb6ff6370382a0b968bea: Status 404 returned error can't find the container with id 21800e133bfe1f6dfcac0efb384a21e202a2aa8e4a7cb6ff6370382a0b968bea Sep 18 18:05:02.213356 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.213327 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:02.213446 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.213390 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:02.213446 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.213427 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:02.942692 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.942657 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9eb368-1450-4221-949e-7707708c92ae" path="/var/lib/kubelet/pods/4a9eb368-1450-4221-949e-7707708c92ae/volumes" Sep 18 18:05:02.943043 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:02.943019 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5405e92-47bf-4a6c-868f-378fd8376557" path="/var/lib/kubelet/pods/f5405e92-47bf-4a6c-868f-378fd8376557/volumes" Sep 18 18:05:03.109566 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:03.109527 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerStarted","Data":"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4"} Sep 18 18:05:03.109759 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:03.109584 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerStarted","Data":"21800e133bfe1f6dfcac0efb384a21e202a2aa8e4a7cb6ff6370382a0b968bea"} Sep 18 18:05:03.157597 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:03.157513 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podStartSLOduration=2.157492694 podStartE2EDuration="2.157492694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:03.157251736 +0000 UTC m=+908.249906842" watchObservedRunningTime="2026-09-18 18:05:03.157492694 +0000 UTC m=+908.250147782" Sep 18 18:05:03.912558 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:03.912526 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:03.917523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:03.917499 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:04.115445 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:04.115417 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:04.116408 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:04.116387 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:09.673174 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.673141 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:09.676194 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.676167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.684441 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.680448 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-284e39c5\"" Sep 18 18:05:09.684441 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.680897 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-284e39c5-dockercfg-pr5bh\"" Sep 18 18:05:09.717429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.717384 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:09.762205 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.762146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwstm\" (UniqueName: \"kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.762510 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.762471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.864966 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.864914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwstm\" (UniqueName: \"kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.865700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.865675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.867101 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.866992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.880043 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.880016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwstm\" (UniqueName: \"kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm\") pod \"payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:09.896566 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.896533 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:09.899179 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.899161 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:09.915345 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.915322 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:09.968718 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.968646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xf5\" (UniqueName: \"kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:09.968718 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.968686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:09.991176 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:09.991149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:10.070298 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.070258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xf5\" (UniqueName: \"kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:10.070466 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.070325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:10.071125 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.071098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:10.088330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.088297 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xf5\" (UniqueName: \"kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5\") pod \"payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:10.210155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.210122 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:10.282116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.282089 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:10.282682 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:10.282646 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5a3892_a20c_4c78_9be0_7be6c7b3609e.slice/crio-e606226ac193add0e5491da4f691b9e69fbded9ca32004dba1ade4e2e761e490 WatchSource:0}: Error finding container e606226ac193add0e5491da4f691b9e69fbded9ca32004dba1ade4e2e761e490: Status 404 returned error can't find the container with id e606226ac193add0e5491da4f691b9e69fbded9ca32004dba1ade4e2e761e490 Sep 18 18:05:10.500319 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:10.500290 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:10.502324 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:10.502300 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623d2d72_78a3_4dfe_865d_e1bc2a0160f9.slice/crio-6c2824047888d0d3107522f45c7a78b277037a32ddc3bc37028ea61ef7b3dae9 WatchSource:0}: Error finding container 6c2824047888d0d3107522f45c7a78b277037a32ddc3bc37028ea61ef7b3dae9: Status 404 returned error can't find the container with id 6c2824047888d0d3107522f45c7a78b277037a32ddc3bc37028ea61ef7b3dae9 Sep 18 18:05:11.161077 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.161034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" event={"ID":"623d2d72-78a3-4dfe-865d-e1bc2a0160f9","Type":"ContainerStarted","Data":"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7"} Sep 18 18:05:11.161521 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.161102 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" event={"ID":"623d2d72-78a3-4dfe-865d-e1bc2a0160f9","Type":"ContainerStarted","Data":"6c2824047888d0d3107522f45c7a78b277037a32ddc3bc37028ea61ef7b3dae9"} Sep 18 18:05:11.161521 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.161229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:11.162568 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.162547 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" event={"ID":"fd5a3892-a20c-4c78-9be0-7be6c7b3609e","Type":"ContainerStarted","Data":"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e"} Sep 18 18:05:11.162672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.162583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" event={"ID":"fd5a3892-a20c-4c78-9be0-7be6c7b3609e","Type":"ContainerStarted","Data":"e606226ac193add0e5491da4f691b9e69fbded9ca32004dba1ade4e2e761e490"} Sep 18 18:05:11.162729 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.162666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:11.197422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.197368 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" podStartSLOduration=2.1973507 podStartE2EDuration="2.1973507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:11.19649433 +0000 UTC m=+916.289149422" watchObservedRunningTime="2026-09-18 18:05:11.1973507 +0000 UTC m=+916.290005788" Sep 18 18:05:11.227079 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:11.227042 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" podStartSLOduration=2.227029452 podStartE2EDuration="2.227029452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:11.225558367 +0000 UTC m=+916.318213474" watchObservedRunningTime="2026-09-18 18:05:11.227029452 +0000 UTC m=+916.319684539" Sep 18 18:05:22.170824 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:22.170791 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:22.171229 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:22.170871 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:28.265363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.265252 2570 generic.go:356] "Generic (PLEG): container finished" podID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerID="8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4" exitCode=0 Sep 18 18:05:28.265363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.265329 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerDied","Data":"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4"} Sep 18 18:05:28.265926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.265886 2570 scope.go:117] "RemoveContainer" containerID="8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4" Sep 18 18:05:28.267042 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.267016 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:28.267101 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.267091 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:28.267137 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:28.267127 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:29.273014 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:29.272978 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerStarted","Data":"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf"} Sep 18 18:05:29.913356 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:29.913320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:29.914482 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:29.914459 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 18 18:05:29.914582 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:29.914515 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 18 18:05:30.912983 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:30.912938 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 18 18:05:30.913506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:30.913009 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 18 18:05:31.913792 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:31.913751 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 18 18:05:31.914329 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:31.913823 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 18 18:05:32.913460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:32.913419 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 18 18:05:32.913664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:32.913483 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 18 18:05:33.912876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:33.912841 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 18 18:05:33.913306 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:33.912919 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 18 18:05:34.115429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:34.115392 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:34.917529 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:34.917498 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:34.918478 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:34.918455 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:35.856225 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:35.856173 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp"] Sep 18 18:05:35.859023 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:35.859006 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:35.862710 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:35.862686 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:05:35.863258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:35.863238 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:05:35.892483 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:35.892449 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp"] Sep 18 18:05:36.040634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:36.040593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp\" (UID: \"7bbf044c-fa9b-469a-81fc-62cbab610d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:36.141476 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:36.141390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp\" (UID: \"7bbf044c-fa9b-469a-81fc-62cbab610d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:36.142030 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:36.142010 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp\" (UID: \"7bbf044c-fa9b-469a-81fc-62cbab610d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:36.169138 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:36.169110 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:36.449982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:36.449958 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp"] Sep 18 18:05:36.451956 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:36.451920 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbf044c_fa9b_469a_81fc_62cbab610d93.slice/crio-53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0 WatchSource:0}: Error finding container 53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0: Status 404 returned error can't find the container with id 53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0 Sep 18 18:05:37.325352 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:37.325320 2570 generic.go:356] "Generic (PLEG): container finished" podID="7bbf044c-fa9b-469a-81fc-62cbab610d93" containerID="e8239cc5d38bb7053a3845b75edfdbb18c5599bd79a1f79d2825af69c18b2d9d" exitCode=0 Sep 18 18:05:37.325766 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:37.325415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" event={"ID":"7bbf044c-fa9b-469a-81fc-62cbab610d93","Type":"ContainerDied","Data":"e8239cc5d38bb7053a3845b75edfdbb18c5599bd79a1f79d2825af69c18b2d9d"} Sep 18 18:05:37.325766 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:37.325465 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" event={"ID":"7bbf044c-fa9b-469a-81fc-62cbab610d93","Type":"ContainerStarted","Data":"53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0"} Sep 18 18:05:38.462927 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:38.462900 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:38.464067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:38.464049 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca\") pod \"7bbf044c-fa9b-469a-81fc-62cbab610d93\" (UID: \"7bbf044c-fa9b-469a-81fc-62cbab610d93\") " Sep 18 18:05:38.464365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:38.464346 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bbf044c-fa9b-469a-81fc-62cbab610d93" (UID: "7bbf044c-fa9b-469a-81fc-62cbab610d93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:38.565553 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:38.565520 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf044c-fa9b-469a-81fc-62cbab610d93-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:39.337706 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:39.337610 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" Sep 18 18:05:39.337706 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:39.337645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp" event={"ID":"7bbf044c-fa9b-469a-81fc-62cbab610d93","Type":"ContainerDied","Data":"53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0"} Sep 18 18:05:39.337706 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:39.337706 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53fed31b82c91e37ce02d1bf3d126f8303e893b27f0aba6d44f37c55e639a6f0" Sep 18 18:05:47.420308 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.420271 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:47.420728 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.420550 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" podUID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" containerName="payload-processing" containerID="cri-o://19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7" gracePeriod=30 Sep 18 18:05:47.457890 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.457852 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:47.458103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.458065 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" podUID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" containerName="payload-pre-processing" containerID="cri-o://2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e" gracePeriod=30 Sep 18 18:05:47.699836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.699809 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:47.762230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.756500 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume\") pod \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " Sep 18 18:05:47.762230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.756573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2xf5\" (UniqueName: \"kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5\") pod \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\" (UID: \"623d2d72-78a3-4dfe-865d-e1bc2a0160f9\") " Sep 18 18:05:47.762230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.757570 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "623d2d72-78a3-4dfe-865d-e1bc2a0160f9" (UID: "623d2d72-78a3-4dfe-865d-e1bc2a0160f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:47.771947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.771888 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5" (OuterVolumeSpecName: "kube-api-access-f2xf5") pod "623d2d72-78a3-4dfe-865d-e1bc2a0160f9" (UID: "623d2d72-78a3-4dfe-865d-e1bc2a0160f9"). InnerVolumeSpecName "kube-api-access-f2xf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:47.781415 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.781382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:47.837237 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.837144 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp"] Sep 18 18:05:47.842918 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.842888 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e39c5-kbfbp"] Sep 18 18:05:47.858027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.858001 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume\") pod \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " Sep 18 18:05:47.858146 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.858129 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwstm\" (UniqueName: \"kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm\") pod \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\" (UID: \"fd5a3892-a20c-4c78-9be0-7be6c7b3609e\") " Sep 18 18:05:47.858472 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.858449 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd5a3892-a20c-4c78-9be0-7be6c7b3609e" (UID: "fd5a3892-a20c-4c78-9be0-7be6c7b3609e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:47.858524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.858468 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:47.858524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.858485 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xf5\" (UniqueName: \"kubernetes.io/projected/623d2d72-78a3-4dfe-865d-e1bc2a0160f9-kube-api-access-f2xf5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:47.860309 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.860283 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm" (OuterVolumeSpecName: "kube-api-access-kwstm") pod "fd5a3892-a20c-4c78-9be0-7be6c7b3609e" (UID: "fd5a3892-a20c-4c78-9be0-7be6c7b3609e"). InnerVolumeSpecName "kube-api-access-kwstm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:47.959847 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.959753 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:47.959847 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:47.959789 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwstm\" (UniqueName: \"kubernetes.io/projected/fd5a3892-a20c-4c78-9be0-7be6c7b3609e-kube-api-access-kwstm\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:48.392590 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.392500 2570 generic.go:356] "Generic (PLEG): container finished" podID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" containerID="19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7" exitCode=0 Sep 18 18:05:48.392590 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.392536 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" Sep 18 18:05:48.392790 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.392589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" event={"ID":"623d2d72-78a3-4dfe-865d-e1bc2a0160f9","Type":"ContainerDied","Data":"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7"} Sep 18 18:05:48.392790 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.392642 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn" event={"ID":"623d2d72-78a3-4dfe-865d-e1bc2a0160f9","Type":"ContainerDied","Data":"6c2824047888d0d3107522f45c7a78b277037a32ddc3bc37028ea61ef7b3dae9"} Sep 18 18:05:48.392790 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.392675 2570 scope.go:117] "RemoveContainer" containerID="19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7" Sep 18 18:05:48.393965 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.393939 2570 generic.go:356] "Generic (PLEG): container finished" podID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" containerID="2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e" exitCode=0 Sep 18 18:05:48.394089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.393975 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" Sep 18 18:05:48.394089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.394011 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" event={"ID":"fd5a3892-a20c-4c78-9be0-7be6c7b3609e","Type":"ContainerDied","Data":"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e"} Sep 18 18:05:48.394089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.394067 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz" event={"ID":"fd5a3892-a20c-4c78-9be0-7be6c7b3609e","Type":"ContainerDied","Data":"e606226ac193add0e5491da4f691b9e69fbded9ca32004dba1ade4e2e761e490"} Sep 18 18:05:48.402060 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.402039 2570 scope.go:117] "RemoveContainer" containerID="19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7" Sep 18 18:05:48.402333 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:48.402313 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7\": container with ID starting with 19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7 not found: ID does not exist" containerID="19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7" Sep 18 18:05:48.402405 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.402340 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7"} err="failed to get container status \"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7\": rpc error: code = NotFound desc = could not find container \"19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7\": container with ID starting with 19681084d57352df1dde317697a221e8dc77a1816433cc7455081757270265b7 not found: ID does not exist" Sep 18 18:05:48.402405 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.402357 2570 scope.go:117] "RemoveContainer" containerID="2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e" Sep 18 18:05:48.410327 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.410311 2570 scope.go:117] "RemoveContainer" containerID="2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e" Sep 18 18:05:48.410575 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:48.410556 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e\": container with ID starting with 2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e not found: ID does not exist" containerID="2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e" Sep 18 18:05:48.410638 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.410584 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e"} err="failed to get container status \"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e\": rpc error: code = NotFound desc = could not find container \"2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e\": container with ID starting with 2a15e379b074e82efa954c39a042fc82fb0e595ed3c64d32f93b3dbba21a1f2e not found: ID does not exist" Sep 18 18:05:48.449548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.449523 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:48.472044 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.472015 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-284e39c5-8c85759d-cq4hz"] Sep 18 18:05:48.509383 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.509360 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:48.511455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.511434 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-284e39c5-599b5d885c-xw6rn"] Sep 18 18:05:48.941642 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.941605 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" path="/var/lib/kubelet/pods/623d2d72-78a3-4dfe-865d-e1bc2a0160f9/volumes" Sep 18 18:05:48.941970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.941957 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bbf044c-fa9b-469a-81fc-62cbab610d93" path="/var/lib/kubelet/pods/7bbf044c-fa9b-469a-81fc-62cbab610d93/volumes" Sep 18 18:05:48.942332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:48.942318 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" path="/var/lib/kubelet/pods/fd5a3892-a20c-4c78-9be0-7be6c7b3609e/volumes" Sep 18 18:05:49.086950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.086908 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:49.087727 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.087629 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" containerID="cri-o://bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf" gracePeriod=30 Sep 18 18:05:49.899821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.899792 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900167 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" containerName="payload-pre-processing" Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900195 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" containerName="payload-pre-processing" Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900220 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" containerName="payload-processing" Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900229 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" containerName="payload-processing" Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900238 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bbf044c-fa9b-469a-81fc-62cbab610d93" containerName="revoke-keys" Sep 18 18:05:49.900252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900244 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bbf044c-fa9b-469a-81fc-62cbab610d93" containerName="revoke-keys" Sep 18 18:05:49.900599 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900306 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="623d2d72-78a3-4dfe-865d-e1bc2a0160f9" containerName="payload-processing" Sep 18 18:05:49.900599 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900314 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bbf044c-fa9b-469a-81fc-62cbab610d93" containerName="revoke-keys" Sep 18 18:05:49.900599 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.900323 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5a3892-a20c-4c78-9be0-7be6c7b3609e" containerName="payload-pre-processing" Sep 18 18:05:49.903617 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.903599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.908836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.908816 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-31e0da22-openshift-default-dockercfg-2rnjc\"" Sep 18 18:05:49.920029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.920007 2570 patch_prober.go:28] interesting pod/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:05:49.920119 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.920057 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:05:49.921333 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.921307 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:05:49.979761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.979912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.979912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.979912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmcn\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.980053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.980053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.979973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.980053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.980004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.980223 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.980111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:49.980223 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:49.980147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.081852 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.081810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.082547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.082523 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.083119 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.082462 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.083375 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.083355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.083664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.083645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.084433 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.083617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.084608 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.084591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.084925 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.084903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmcn\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.085165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.085146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.085422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.085404 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.085691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.085666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.085968 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.085467 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.086104 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.085832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.086579 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.086554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.089884 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.089852 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.091252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.091231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.102203 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.098337 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.102203 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.100281 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmcn\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn\") pod \"e2e-derive-31e0da22-openshift-default-659679996-s5dmc\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.213304 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.213270 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:50.498598 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.498576 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:05:50.500357 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:50.500328 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafaafb8_1418_40fe_b71d_26008c7de6ca.slice/crio-73909d5639de51e589e71bc24ff3f59473b7534446a77709d1fa0908b1e5a28a WatchSource:0}: Error finding container 73909d5639de51e589e71bc24ff3f59473b7534446a77709d1fa0908b1e5a28a: Status 404 returned error can't find the container with id 73909d5639de51e589e71bc24ff3f59473b7534446a77709d1fa0908b1e5a28a Sep 18 18:05:50.502541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.502510 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:50.502640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.502573 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:50.502640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:50.502601 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:05:51.413198 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:51.413159 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerStarted","Data":"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea"} Sep 18 18:05:51.413585 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:51.413246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerStarted","Data":"73909d5639de51e589e71bc24ff3f59473b7534446a77709d1fa0908b1e5a28a"} Sep 18 18:05:51.466327 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:51.466268 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podStartSLOduration=2.46625276 podStartE2EDuration="2.46625276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:51.464157549 +0000 UTC m=+956.556812657" watchObservedRunningTime="2026-09-18 18:05:51.46625276 +0000 UTC m=+956.558907848" Sep 18 18:05:52.213671 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:52.213635 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:52.218814 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:52.218787 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:52.419338 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:52.419306 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:52.420412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:52.420393 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:05:54.345034 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.345004 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:54.431978 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.431945 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432010 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432067 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432094 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432130 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432202 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432172 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432469 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432228 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432469 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432278 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432469 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432309 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtcwd\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd\") pod \"7403d04a-361e-4e6c-bc0f-298efe916f38\" (UID: \"7403d04a-361e-4e6c-bc0f-298efe916f38\") " Sep 18 18:05:54.432626 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432480 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:05:54.432626 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432514 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data" (OuterVolumeSpecName: "istio-data") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:05:54.432626 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.432615 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:05:54.433100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.433063 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403d04a-361e-4e6c-bc0f-298efe916f38-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.433100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.433086 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.433100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.433098 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.433384 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.433145 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:05:54.433384 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.433301 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:05:54.434982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.434956 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd" (OuterVolumeSpecName: "kube-api-access-qtcwd") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "kube-api-access-qtcwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:54.435100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.434996 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token" (OuterVolumeSpecName: "istio-token") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:05:54.435250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435226 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:05:54.435440 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435422 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7403d04a-361e-4e6c-bc0f-298efe916f38" (UID: "7403d04a-361e-4e6c-bc0f-298efe916f38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:05:54.435849 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435827 2570 generic.go:356] "Generic (PLEG): container finished" podID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerID="bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf" exitCode=0 Sep 18 18:05:54.435920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435875 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" Sep 18 18:05:54.435988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerDied","Data":"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf"} Sep 18 18:05:54.436036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.435992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2" event={"ID":"7403d04a-361e-4e6c-bc0f-298efe916f38","Type":"ContainerDied","Data":"21800e133bfe1f6dfcac0efb384a21e202a2aa8e4a7cb6ff6370382a0b968bea"} Sep 18 18:05:54.436036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.436015 2570 scope.go:117] "RemoveContainer" containerID="bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf" Sep 18 18:05:54.454338 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.454315 2570 scope.go:117] "RemoveContainer" containerID="8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4" Sep 18 18:05:54.462977 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.462961 2570 scope.go:117] "RemoveContainer" containerID="bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf" Sep 18 18:05:54.463268 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:54.463249 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf\": container with ID starting with bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf not found: ID does not exist" containerID="bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf" Sep 18 18:05:54.463314 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.463278 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf"} err="failed to get container status \"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf\": rpc error: code = NotFound desc = could not find container \"bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf\": container with ID starting with bac8dbee515816831be28ecd0333efe232bee86c65816f2fdb14cce77ba854cf not found: ID does not exist" Sep 18 18:05:54.463314 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.463296 2570 scope.go:117] "RemoveContainer" containerID="8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4" Sep 18 18:05:54.463560 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:05:54.463540 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4\": container with ID starting with 8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4 not found: ID does not exist" containerID="8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4" Sep 18 18:05:54.463603 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.463566 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4"} err="failed to get container status \"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4\": rpc error: code = NotFound desc = could not find container \"8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4\": container with ID starting with 8ecf07d59c62d841a93864fecac9caecd5e382a4b3af64a4e1c0d9845d5c18a4 not found: ID does not exist" Sep 18 18:05:54.479883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.479842 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:54.486704 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.486678 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-284e39c5-openshift-default-6f6b8f8dd6-28zv2"] Sep 18 18:05:54.534311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534277 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.534311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534306 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcwd\" (UniqueName: \"kubernetes.io/projected/7403d04a-361e-4e6c-bc0f-298efe916f38-kube-api-access-qtcwd\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.534311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534317 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.534597 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534327 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.534597 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534336 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.534597 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.534344 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403d04a-361e-4e6c-bc0f-298efe916f38-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:05:54.941847 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:54.941814 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" path="/var/lib/kubelet/pods/7403d04a-361e-4e6c-bc0f-298efe916f38/volumes" Sep 18 18:05:57.438471 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438439 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:05:57.438850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438802 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.438850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438814 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.438850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438825 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.438850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438831 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.439037 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.438899 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.442073 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.442053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.444523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.444501 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-31e0da22\"" Sep 18 18:05:57.444607 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.444596 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-31e0da22-dockercfg-4s5xl\"" Sep 18 18:05:57.457456 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.457431 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:05:57.457823 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.457810 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7403d04a-361e-4e6c-bc0f-298efe916f38" containerName="istio-proxy" Sep 18 18:05:57.459739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.459724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.491019 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.488788 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:05:57.508069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.507321 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:05:57.559537 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.559496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fx2\" (UniqueName: \"kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.559709 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.559562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.559709 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.559617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.559709 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.559697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjzg\" (UniqueName: \"kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.660838 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.660805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.660982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.660871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjzg\" (UniqueName: \"kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.660982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.660908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fx2\" (UniqueName: \"kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.660982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.660927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.661551 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.661532 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.661662 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.661645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.670761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.670728 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjzg\" (UniqueName: \"kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg\") pod \"payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:57.678976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.678943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fx2\" (UniqueName: \"kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2\") pod \"payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.752734 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.752639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:57.769451 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:57.769414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:58.035883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.035806 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9"] Sep 18 18:05:58.038562 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.038544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:05:58.042504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.042478 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:05:58.042989 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.042955 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:05:58.045799 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.045770 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:05:58.047434 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:58.047411 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f37dc3_d5a5_47f1_b732_ef396b054ff5.slice/crio-2271b751df9916deafd1a4b97abca7ba129c403ed4ad8ea6533bb30a369ee57b WatchSource:0}: Error finding container 2271b751df9916deafd1a4b97abca7ba129c403ed4ad8ea6533bb30a369ee57b: Status 404 returned error can't find the container with id 2271b751df9916deafd1a4b97abca7ba129c403ed4ad8ea6533bb30a369ee57b Sep 18 18:05:58.069421 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.069388 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:05:58.072309 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.072287 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9"] Sep 18 18:05:58.076341 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:58.076316 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac03eb74_956c_4d81_b8be_f29dce811c0b.slice/crio-c12268e9f11ef5173fc5c3adcb5bb951f874cc345a78da4f92dcc67eaf127242 WatchSource:0}: Error finding container c12268e9f11ef5173fc5c3adcb5bb951f874cc345a78da4f92dcc67eaf127242: Status 404 returned error can't find the container with id c12268e9f11ef5173fc5c3adcb5bb951f874cc345a78da4f92dcc67eaf127242 Sep 18 18:05:58.164736 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.164702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a39784-rjxm9\" (UID: \"2d761551-162c-4129-b072-563b43537862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:05:58.265658 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.265626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a39784-rjxm9\" (UID: \"2d761551-162c-4129-b072-563b43537862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:05:58.266278 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.266258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a39784-rjxm9\" (UID: \"2d761551-162c-4129-b072-563b43537862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:05:58.350175 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.350083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:05:58.461865 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.461832 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" event={"ID":"51f37dc3-d5a5-47f1-b732-ef396b054ff5","Type":"ContainerStarted","Data":"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116"} Sep 18 18:05:58.462339 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.461897 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:05:58.462339 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.461924 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" event={"ID":"51f37dc3-d5a5-47f1-b732-ef396b054ff5","Type":"ContainerStarted","Data":"2271b751df9916deafd1a4b97abca7ba129c403ed4ad8ea6533bb30a369ee57b"} Sep 18 18:05:58.463509 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.463487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" event={"ID":"ac03eb74-956c-4d81-b8be-f29dce811c0b","Type":"ContainerStarted","Data":"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1"} Sep 18 18:05:58.463625 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.463526 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" event={"ID":"ac03eb74-956c-4d81-b8be-f29dce811c0b","Type":"ContainerStarted","Data":"c12268e9f11ef5173fc5c3adcb5bb951f874cc345a78da4f92dcc67eaf127242"} Sep 18 18:05:58.463625 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.463551 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:05:58.489120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.489076 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" podStartSLOduration=1.489061065 podStartE2EDuration="1.489061065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:58.487526595 +0000 UTC m=+963.580181707" watchObservedRunningTime="2026-09-18 18:05:58.489061065 +0000 UTC m=+963.581716152" Sep 18 18:05:58.523818 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.523771 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" podStartSLOduration=1.523756288 podStartE2EDuration="1.523756288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:58.523016969 +0000 UTC m=+963.615672079" watchObservedRunningTime="2026-09-18 18:05:58.523756288 +0000 UTC m=+963.616411375" Sep 18 18:05:58.640178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:58.640152 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9"] Sep 18 18:05:58.641878 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:05:58.641853 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d761551_162c_4129_b072_563b43537862.slice/crio-55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733 WatchSource:0}: Error finding container 55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733: Status 404 returned error can't find the container with id 55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733 Sep 18 18:05:59.470813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:59.470781 2570 generic.go:356] "Generic (PLEG): container finished" podID="2d761551-162c-4129-b072-563b43537862" containerID="5ae51465463797ece3d673c756acd975b84d6327c81c738b5fd2a28ff4104a00" exitCode=0 Sep 18 18:05:59.471287 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:59.470847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" event={"ID":"2d761551-162c-4129-b072-563b43537862","Type":"ContainerDied","Data":"5ae51465463797ece3d673c756acd975b84d6327c81c738b5fd2a28ff4104a00"} Sep 18 18:05:59.471287 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:05:59.470882 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" event={"ID":"2d761551-162c-4129-b072-563b43537862","Type":"ContainerStarted","Data":"55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733"} Sep 18 18:06:00.613343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:00.613315 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:06:00.688018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:00.687984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca\") pod \"2d761551-162c-4129-b072-563b43537862\" (UID: \"2d761551-162c-4129-b072-563b43537862\") " Sep 18 18:06:00.688392 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:00.688369 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d761551-162c-4129-b072-563b43537862" (UID: "2d761551-162c-4129-b072-563b43537862"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:06:00.789404 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:00.789314 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d761551-162c-4129-b072-563b43537862-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:01.487336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:01.487308 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" Sep 18 18:06:01.487522 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:01.487354 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9" event={"ID":"2d761551-162c-4129-b072-563b43537862","Type":"ContainerDied","Data":"55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733"} Sep 18 18:06:01.487522 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:01.487409 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d1f3d11f8c542c4db2c7ce654e13a741e36411d868897ce4b91a2bdf8f1733" Sep 18 18:06:09.130082 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:09.130049 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9"] Sep 18 18:06:09.134663 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:09.134636 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a39784-rjxm9"] Sep 18 18:06:09.473181 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:09.473150 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:06:09.473364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:09.473322 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:06:10.942080 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:10.942041 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d761551-162c-4129-b072-563b43537862" path="/var/lib/kubelet/pods/2d761551-162c-4129-b072-563b43537862/volumes" Sep 18 18:06:17.809118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.809079 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht"] Sep 18 18:06:17.809637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.809468 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d761551-162c-4129-b072-563b43537862" containerName="revoke-keys" Sep 18 18:06:17.809637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.809481 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d761551-162c-4129-b072-563b43537862" containerName="revoke-keys" Sep 18 18:06:17.809637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.809543 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d761551-162c-4129-b072-563b43537862" containerName="revoke-keys" Sep 18 18:06:17.812555 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.812532 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:17.815511 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.815491 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:06:17.815511 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.815514 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:06:17.829027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.828998 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht"] Sep 18 18:06:17.850925 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.850894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e0da22-f9mht\" (UID: \"de24e945-9291-4789-8df7-167674129e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:17.952248 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.952209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e0da22-f9mht\" (UID: \"de24e945-9291-4789-8df7-167674129e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:17.952812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:17.952789 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e0da22-f9mht\" (UID: \"de24e945-9291-4789-8df7-167674129e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:18.123270 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:18.123171 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:18.413687 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:18.413661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht"] Sep 18 18:06:18.415625 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:06:18.415597 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde24e945_9291_4789_8df7_167674129e16.slice/crio-0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5 WatchSource:0}: Error finding container 0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5: Status 404 returned error can't find the container with id 0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5 Sep 18 18:06:18.585386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:18.585346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" event={"ID":"de24e945-9291-4789-8df7-167674129e16","Type":"ContainerStarted","Data":"0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5"} Sep 18 18:06:19.593634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.593541 2570 generic.go:356] "Generic (PLEG): container finished" podID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerID="8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea" exitCode=0 Sep 18 18:06:19.593634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.593614 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerDied","Data":"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea"} Sep 18 18:06:19.594122 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.594104 2570 scope.go:117] "RemoveContainer" containerID="8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea" Sep 18 18:06:19.595372 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.595348 2570 generic.go:356] "Generic (PLEG): container finished" podID="de24e945-9291-4789-8df7-167674129e16" containerID="42b33c9351da3b2cf3d93f2a87f0e4c935fdc8b7ed69242cbfe833321f28142f" exitCode=0 Sep 18 18:06:19.595460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.595374 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" event={"ID":"de24e945-9291-4789-8df7-167674129e16","Type":"ContainerDied","Data":"42b33c9351da3b2cf3d93f2a87f0e4c935fdc8b7ed69242cbfe833321f28142f"} Sep 18 18:06:19.595460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.595371 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:19.595540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.595471 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:19.595540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:19.595502 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:20.610403 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:20.610272 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerStarted","Data":"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551"} Sep 18 18:06:20.764671 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:20.764646 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:20.772975 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:20.772952 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca\") pod \"de24e945-9291-4789-8df7-167674129e16\" (UID: \"de24e945-9291-4789-8df7-167674129e16\") " Sep 18 18:06:20.773311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:20.773283 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de24e945-9291-4789-8df7-167674129e16" (UID: "de24e945-9291-4789-8df7-167674129e16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:06:20.874547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:20.874466 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de24e945-9291-4789-8df7-167674129e16-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:21.214346 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.214307 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:21.215452 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.215428 2570 patch_prober.go:28] interesting pod/e2e-derive-31e0da22-openshift-default-659679996-s5dmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 18 18:06:21.215512 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.215483 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 18 18:06:21.617880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.617788 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" Sep 18 18:06:21.617880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.617811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht" event={"ID":"de24e945-9291-4789-8df7-167674129e16","Type":"ContainerDied","Data":"0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5"} Sep 18 18:06:21.617880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:21.617862 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ecaffb723544b8dcc5a10b5ddef5df3de5139796d32ff9b0e1e92e33b0912a5" Sep 18 18:06:22.214412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:22.214371 2570 patch_prober.go:28] interesting pod/e2e-derive-31e0da22-openshift-default-659679996-s5dmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 18 18:06:22.214605 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:22.214448 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 18 18:06:22.419897 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:22.419856 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:23.213992 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:23.213953 2570 patch_prober.go:28] interesting pod/e2e-derive-31e0da22-openshift-default-659679996-s5dmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 18 18:06:23.214514 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:23.214017 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 18 18:06:24.214604 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:24.214567 2570 patch_prober.go:28] interesting pod/e2e-derive-31e0da22-openshift-default-659679996-s5dmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 18 18:06:24.215063 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:24.214640 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 18 18:06:25.214168 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.214129 2570 patch_prober.go:28] interesting pod/e2e-derive-31e0da22-openshift-default-659679996-s5dmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 18 18:06:25.214387 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.214207 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 18 18:06:25.884910 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.884874 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:06:25.885323 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.885149 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" podUID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" containerName="payload-processing" containerID="cri-o://e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116" gracePeriod=30 Sep 18 18:06:25.906035 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.905993 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:06:25.906422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:25.906367 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" podUID="ac03eb74-956c-4d81-b8be-f29dce811c0b" containerName="payload-pre-processing" containerID="cri-o://33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1" gracePeriod=30 Sep 18 18:06:26.180969 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.180947 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:06:26.183455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.183435 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:06:26.218396 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.218368 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:26.225261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.223174 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.235633 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume\") pod \"ac03eb74-956c-4d81-b8be-f29dce811c0b\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.235678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume\") pod \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.235733 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4fx2\" (UniqueName: \"kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2\") pod \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\" (UID: \"51f37dc3-d5a5-47f1-b732-ef396b054ff5\") " Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.235817 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vjzg\" (UniqueName: \"kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg\") pod \"ac03eb74-956c-4d81-b8be-f29dce811c0b\" (UID: \"ac03eb74-956c-4d81-b8be-f29dce811c0b\") " Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.236214 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac03eb74-956c-4d81-b8be-f29dce811c0b" (UID: "ac03eb74-956c-4d81-b8be-f29dce811c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.236327 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac03eb74-956c-4d81-b8be-f29dce811c0b-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:26.238353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.236538 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51f37dc3-d5a5-47f1-b732-ef396b054ff5" (UID: "51f37dc3-d5a5-47f1-b732-ef396b054ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:06:26.239326 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.239284 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg" (OuterVolumeSpecName: "kube-api-access-8vjzg") pod "ac03eb74-956c-4d81-b8be-f29dce811c0b" (UID: "ac03eb74-956c-4d81-b8be-f29dce811c0b"). InnerVolumeSpecName "kube-api-access-8vjzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:06:26.260222 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.259393 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2" (OuterVolumeSpecName: "kube-api-access-b4fx2") pod "51f37dc3-d5a5-47f1-b732-ef396b054ff5" (UID: "51f37dc3-d5a5-47f1-b732-ef396b054ff5"). InnerVolumeSpecName "kube-api-access-b4fx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:06:26.337864 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.337819 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4fx2\" (UniqueName: \"kubernetes.io/projected/51f37dc3-d5a5-47f1-b732-ef396b054ff5-kube-api-access-b4fx2\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:26.338039 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.337952 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjzg\" (UniqueName: \"kubernetes.io/projected/ac03eb74-956c-4d81-b8be-f29dce811c0b-kube-api-access-8vjzg\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:26.338039 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.337984 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f37dc3-d5a5-47f1-b732-ef396b054ff5-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:26.399030 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.398994 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht"] Sep 18 18:06:26.412644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.412567 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e0da22-f9mht"] Sep 18 18:06:26.575822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.575782 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:06:26.652026 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.651991 2570 generic.go:356] "Generic (PLEG): container finished" podID="ac03eb74-956c-4d81-b8be-f29dce811c0b" containerID="33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1" exitCode=0 Sep 18 18:06:26.652212 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.652035 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" Sep 18 18:06:26.652212 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.652074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" event={"ID":"ac03eb74-956c-4d81-b8be-f29dce811c0b","Type":"ContainerDied","Data":"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1"} Sep 18 18:06:26.652212 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.652133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6" event={"ID":"ac03eb74-956c-4d81-b8be-f29dce811c0b","Type":"ContainerDied","Data":"c12268e9f11ef5173fc5c3adcb5bb951f874cc345a78da4f92dcc67eaf127242"} Sep 18 18:06:26.652212 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.652157 2570 scope.go:117] "RemoveContainer" containerID="33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1" Sep 18 18:06:26.653546 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.653520 2570 generic.go:356] "Generic (PLEG): container finished" podID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" containerID="e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116" exitCode=0 Sep 18 18:06:26.653655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.653570 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" Sep 18 18:06:26.653655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.653595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" event={"ID":"51f37dc3-d5a5-47f1-b732-ef396b054ff5","Type":"ContainerDied","Data":"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116"} Sep 18 18:06:26.653655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.653633 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c" event={"ID":"51f37dc3-d5a5-47f1-b732-ef396b054ff5","Type":"ContainerDied","Data":"2271b751df9916deafd1a4b97abca7ba129c403ed4ad8ea6533bb30a369ee57b"} Sep 18 18:06:26.654024 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.653994 2570 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" secret="" err="secret \"e2e-derive-31e0da22-openshift-default-dockercfg-2rnjc\" not found" Sep 18 18:06:26.667505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.667392 2570 scope.go:117] "RemoveContainer" containerID="33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1" Sep 18 18:06:26.668249 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:06:26.668063 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1\": container with ID starting with 33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1 not found: ID does not exist" containerID="33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1" Sep 18 18:06:26.668249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.668114 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1"} err="failed to get container status \"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1\": rpc error: code = NotFound desc = could not find container \"33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1\": container with ID starting with 33a8388e628f2846c330e818cd971755b892babfd5b7514ee5b0368f9d9e9eb1 not found: ID does not exist" Sep 18 18:06:26.668249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.668138 2570 scope.go:117] "RemoveContainer" containerID="e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116" Sep 18 18:06:26.687595 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.687321 2570 scope.go:117] "RemoveContainer" containerID="e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116" Sep 18 18:06:26.688066 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:06:26.687985 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116\": container with ID starting with e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116 not found: ID does not exist" containerID="e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116" Sep 18 18:06:26.688066 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.688022 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116"} err="failed to get container status \"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116\": rpc error: code = NotFound desc = could not find container \"e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116\": container with ID starting with e113389d88ded6699eea0cbb2720fb98f4dff1ff4b4b77f887b22a4d0a32f116 not found: ID does not exist" Sep 18 18:06:26.714896 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.714862 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:06:26.735324 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.735293 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e0da22-6d868b58df-22l7c"] Sep 18 18:06:26.757871 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.757839 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:06:26.779876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.779839 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e0da22-75fcddf7d-jbfc6"] Sep 18 18:06:26.942225 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.942119 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" path="/var/lib/kubelet/pods/51f37dc3-d5a5-47f1-b732-ef396b054ff5/volumes" Sep 18 18:06:26.942630 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.942610 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac03eb74-956c-4d81-b8be-f29dce811c0b" path="/var/lib/kubelet/pods/ac03eb74-956c-4d81-b8be-f29dce811c0b/volumes" Sep 18 18:06:26.942940 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:26.942927 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de24e945-9291-4789-8df7-167674129e16" path="/var/lib/kubelet/pods/de24e945-9291-4789-8df7-167674129e16/volumes" Sep 18 18:06:27.660891 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:27.660857 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" containerID="cri-o://8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551" gracePeriod=30 Sep 18 18:06:32.912914 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:32.912880 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:33.007004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.006910 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.006960 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.006991 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007022 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007069 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007091 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007124 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007166 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjmcn\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007213 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket\") pod \"fafaafb8-1418-40fe-b71d-26008c7de6ca\" (UID: \"fafaafb8-1418-40fe-b71d-26008c7de6ca\") " Sep 18 18:06:33.007347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007325 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:06:33.007716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007389 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:06:33.007716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007484 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:06:33.007716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:06:33.007874 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007810 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:06:33.007958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007938 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.008016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007963 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafaafb8-1418-40fe-b71d-26008c7de6ca-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.008016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007978 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.008016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.007991 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.008016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.008004 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.009796 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.009770 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn" (OuterVolumeSpecName: "kube-api-access-xjmcn") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "kube-api-access-xjmcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:06:33.009885 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.009795 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:06:33.009885 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.009812 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:06:33.009885 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.009813 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "fafaafb8-1418-40fe-b71d-26008c7de6ca" (UID: "fafaafb8-1418-40fe-b71d-26008c7de6ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:06:33.108909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.108878 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.108909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.108906 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.108909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.108917 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjmcn\" (UniqueName: \"kubernetes.io/projected/fafaafb8-1418-40fe-b71d-26008c7de6ca-kube-api-access-xjmcn\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.109132 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.108925 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafaafb8-1418-40fe-b71d-26008c7de6ca-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:06:33.697410 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.697373 2570 generic.go:356] "Generic (PLEG): container finished" podID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerID="8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551" exitCode=0 Sep 18 18:06:33.697618 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.697442 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" Sep 18 18:06:33.697618 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.697465 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerDied","Data":"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551"} Sep 18 18:06:33.697618 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.697522 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc" event={"ID":"fafaafb8-1418-40fe-b71d-26008c7de6ca","Type":"ContainerDied","Data":"73909d5639de51e589e71bc24ff3f59473b7534446a77709d1fa0908b1e5a28a"} Sep 18 18:06:33.697618 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.697546 2570 scope.go:117] "RemoveContainer" containerID="8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551" Sep 18 18:06:33.707883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.707864 2570 scope.go:117] "RemoveContainer" containerID="8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea" Sep 18 18:06:33.716074 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.716051 2570 scope.go:117] "RemoveContainer" containerID="8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551" Sep 18 18:06:33.716423 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:06:33.716402 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551\": container with ID starting with 8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551 not found: ID does not exist" containerID="8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551" Sep 18 18:06:33.716518 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.716436 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551"} err="failed to get container status \"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551\": rpc error: code = NotFound desc = could not find container \"8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551\": container with ID starting with 8d6d8d183d5a9b447f91f3338b6d6c7e66f71c28d74ea99d6706fc205961b551 not found: ID does not exist" Sep 18 18:06:33.716518 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.716462 2570 scope.go:117] "RemoveContainer" containerID="8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea" Sep 18 18:06:33.716775 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:06:33.716734 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea\": container with ID starting with 8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea not found: ID does not exist" containerID="8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea" Sep 18 18:06:33.716825 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.716781 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea"} err="failed to get container status \"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea\": rpc error: code = NotFound desc = could not find container \"8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea\": container with ID starting with 8872920660f26900749118162eca6a6b1da67b52673a41728490298dd6e0bfea not found: ID does not exist" Sep 18 18:06:33.731898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.731875 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:06:33.742086 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:33.742061 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-31e0da22-openshift-default-659679996-s5dmc"] Sep 18 18:06:34.941853 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:34.941819 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" path="/var/lib/kubelet/pods/fafaafb8-1418-40fe-b71d-26008c7de6ca/volumes" Sep 18 18:06:53.731892 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.731856 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732253 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732271 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732297 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de24e945-9291-4789-8df7-167674129e16" containerName="revoke-keys" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732305 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="de24e945-9291-4789-8df7-167674129e16" containerName="revoke-keys" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732325 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" containerName="payload-processing" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732334 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" containerName="payload-processing" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732353 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac03eb74-956c-4d81-b8be-f29dce811c0b" containerName="payload-pre-processing" Sep 18 18:06:53.732369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732362 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac03eb74-956c-4d81-b8be-f29dce811c0b" containerName="payload-pre-processing" Sep 18 18:06:53.732756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732436 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="de24e945-9291-4789-8df7-167674129e16" containerName="revoke-keys" Sep 18 18:06:53.732756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732449 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:06:53.732756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732458 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:06:53.732756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732468 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac03eb74-956c-4d81-b8be-f29dce811c0b" containerName="payload-pre-processing" Sep 18 18:06:53.732756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.732479 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f37dc3-d5a5-47f1-b732-ef396b054ff5" containerName="payload-processing" Sep 18 18:06:53.735708 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.735688 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.738839 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.738817 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ef5655-openshift-default-dockercfg-bzk65\"" Sep 18 18:06:53.764109 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.764085 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:06:53.910354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lpd\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910538 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:53.910666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:53.910641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.011883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.011883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.011883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.011982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012103 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lpd\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012325 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012634 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012612 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012712 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012650 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.012712 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.012685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.014272 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.014251 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.014416 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.014398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.028148 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.028130 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.035641 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.035613 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lpd\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd\") pod \"e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.046503 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.046480 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:54.340575 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.340544 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:06:54.342361 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:06:54.342332 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2055ee3_eedc_41b2_a916_8996011607b6.slice/crio-167ecc67725b90d69d4c86b010ba138351151b8dee372bb52474e69b55dd44ff WatchSource:0}: Error finding container 167ecc67725b90d69d4c86b010ba138351151b8dee372bb52474e69b55dd44ff: Status 404 returned error can't find the container with id 167ecc67725b90d69d4c86b010ba138351151b8dee372bb52474e69b55dd44ff Sep 18 18:06:54.344492 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.344463 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:54.344584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.344523 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:54.344584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.344552 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:06:54.819957 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.819919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerStarted","Data":"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47"} Sep 18 18:06:54.820387 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.820012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerStarted","Data":"167ecc67725b90d69d4c86b010ba138351151b8dee372bb52474e69b55dd44ff"} Sep 18 18:06:54.880423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:54.880371 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podStartSLOduration=1.880355724 podStartE2EDuration="1.880355724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:06:54.878883389 +0000 UTC m=+1019.971538499" watchObservedRunningTime="2026-09-18 18:06:54.880355724 +0000 UTC m=+1019.973010811" Sep 18 18:06:55.047439 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:55.047402 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:55.052427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:55.052403 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:55.825440 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:55.825402 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:06:55.826266 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:06:55.826248 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:07:02.773855 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.773816 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:07:02.774299 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.774283 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:07:02.774348 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.774301 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fafaafb8-1418-40fe-b71d-26008c7de6ca" containerName="istio-proxy" Sep 18 18:07:02.778964 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.778944 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.792495 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.792474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ef5655\"" Sep 18 18:07:02.810124 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.810097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:07:02.853398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.853372 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:07:02.856641 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.856626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:02.859788 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.859767 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ef5655-dockercfg-96ln8\"" Sep 18 18:07:02.877650 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.877627 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:07:02.894856 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.894832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.894972 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.894868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgk4\" (UniqueName: \"kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.894972 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.894943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcr5\" (UniqueName: \"kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:02.895042 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.895014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:02.995828 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.995792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.995828 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.995840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgk4\" (UniqueName: \"kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.996062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.995881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcr5\" (UniqueName: \"kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:02.996062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.995929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:02.996570 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.996546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:02.996656 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:02.996638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:03.010291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.010267 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgk4\" (UniqueName: \"kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4\") pod \"payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:03.010453 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.010435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcr5\" (UniqueName: \"kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5\") pod \"payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:03.089303 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.089212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:03.166542 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.166509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:03.387250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.387155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:07:03.393243 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:07:03.393143 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56312c7a_a2b1_4872_a1c5_b8fc1c9fd469.slice/crio-9e2d0e1e69fc5f75651c6b7b537fd7f1083599e8ef68192ecfa0936d94c7236e WatchSource:0}: Error finding container 9e2d0e1e69fc5f75651c6b7b537fd7f1083599e8ef68192ecfa0936d94c7236e: Status 404 returned error can't find the container with id 9e2d0e1e69fc5f75651c6b7b537fd7f1083599e8ef68192ecfa0936d94c7236e Sep 18 18:07:03.447484 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.447442 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:07:03.448850 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:07:03.448824 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8ed9a2_fbc7_4096_8271_cebe2cf36122.slice/crio-522a85fdd85f963091c0d5e2c8d8376e7d59af1b9ae539b17ca8051f6b312998 WatchSource:0}: Error finding container 522a85fdd85f963091c0d5e2c8d8376e7d59af1b9ae539b17ca8051f6b312998: Status 404 returned error can't find the container with id 522a85fdd85f963091c0d5e2c8d8376e7d59af1b9ae539b17ca8051f6b312998 Sep 18 18:07:03.873006 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.872968 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" event={"ID":"fe8ed9a2-fbc7-4096-8271-cebe2cf36122","Type":"ContainerStarted","Data":"424f2cffa15acc6f79b49ae5b03e4fdfd4c6155521e2baad7ff107f4fc0606aa"} Sep 18 18:07:03.873520 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.873034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" event={"ID":"fe8ed9a2-fbc7-4096-8271-cebe2cf36122","Type":"ContainerStarted","Data":"522a85fdd85f963091c0d5e2c8d8376e7d59af1b9ae539b17ca8051f6b312998"} Sep 18 18:07:03.873520 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.873070 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:03.874616 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.874584 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" event={"ID":"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469","Type":"ContainerStarted","Data":"e031882f8b030148426539572eeeff6132f1547222a64cbb3ff5994f6cfa2489"} Sep 18 18:07:03.874746 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.874637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:03.874746 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.874662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" event={"ID":"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469","Type":"ContainerStarted","Data":"9e2d0e1e69fc5f75651c6b7b537fd7f1083599e8ef68192ecfa0936d94c7236e"} Sep 18 18:07:03.896577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.896534 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" podStartSLOduration=1.896518785 podStartE2EDuration="1.896518785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:03.894016126 +0000 UTC m=+1028.986671227" watchObservedRunningTime="2026-09-18 18:07:03.896518785 +0000 UTC m=+1028.989173872" Sep 18 18:07:03.921501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:03.921447 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" podStartSLOduration=1.921429474 podStartE2EDuration="1.921429474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:03.91920359 +0000 UTC m=+1029.011858704" watchObservedRunningTime="2026-09-18 18:07:03.921429474 +0000 UTC m=+1029.014084543" Sep 18 18:07:14.883078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:14.883044 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:07:14.883562 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:14.883127 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:07:19.968015 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.967980 2570 generic.go:356] "Generic (PLEG): container finished" podID="f2055ee3-eedc-41b2-a916-8996011607b6" containerID="c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47" exitCode=0 Sep 18 18:07:19.968452 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.968058 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerDied","Data":"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47"} Sep 18 18:07:19.968559 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.968543 2570 scope.go:117] "RemoveContainer" containerID="c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47" Sep 18 18:07:19.969729 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.969703 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:07:19.969794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.969769 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:07:19.969833 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:19.969801 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:07:20.975258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:20.975219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerStarted","Data":"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f"} Sep 18 18:07:21.047344 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:21.047310 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:07:21.048395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:21.048364 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:21.048525 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:21.048419 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:22.047592 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:22.047555 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:22.047972 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:22.047611 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:23.047436 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:23.047397 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:23.047615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:23.047475 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:24.047199 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:24.047152 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:24.047379 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:24.047264 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:25.047781 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:25.047740 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:25.048208 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:25.047814 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:25.825772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:25.825735 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:07:26.047217 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:26.047161 2570 patch_prober.go:28] interesting pod/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 18 18:07:26.047420 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:26.047242 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 18 18:07:27.051236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:27.051170 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:07:27.052450 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:07:27.052429 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:08:38.265011 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.264966 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:08:38.267782 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.267759 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.271117 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.271097 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3ffa634f\"" Sep 18 18:08:38.271915 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.271898 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3ffa634f-dockercfg-nz6dm\"" Sep 18 18:08:38.289066 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.289042 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:08:38.338601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.338573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2d5\" (UniqueName: \"kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.338779 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.338634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.440116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.440082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.440316 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.440228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2d5\" (UniqueName: \"kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.440875 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.440855 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.452789 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.452757 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2d5\" (UniqueName: \"kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5\") pod \"payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.577765 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.577684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:38.907803 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:38.907777 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:08:38.909872 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:08:38.909844 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d307f29_0800_4630_a81e_9c59a7702f62.slice/crio-22ebff3246ff6988c6acd7f51d5809d3b35ddb6d8d707c7555a5a3bb5bf8b4c0 WatchSource:0}: Error finding container 22ebff3246ff6988c6acd7f51d5809d3b35ddb6d8d707c7555a5a3bb5bf8b4c0: Status 404 returned error can't find the container with id 22ebff3246ff6988c6acd7f51d5809d3b35ddb6d8d707c7555a5a3bb5bf8b4c0 Sep 18 18:08:39.427082 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:39.427026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" event={"ID":"6d307f29-0800-4630-a81e-9c59a7702f62","Type":"ContainerStarted","Data":"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94"} Sep 18 18:08:39.427516 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:39.427110 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:08:39.427516 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:39.427134 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" event={"ID":"6d307f29-0800-4630-a81e-9c59a7702f62","Type":"ContainerStarted","Data":"22ebff3246ff6988c6acd7f51d5809d3b35ddb6d8d707c7555a5a3bb5bf8b4c0"} Sep 18 18:08:39.469300 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:39.469235 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" podStartSLOduration=1.469213925 podStartE2EDuration="1.469213925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:08:39.466361772 +0000 UTC m=+1124.559016890" watchObservedRunningTime="2026-09-18 18:08:39.469213925 +0000 UTC m=+1124.561869013" Sep 18 18:08:50.434928 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:08:50.434897 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:09:11.845858 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.845775 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5"] Sep 18 18:09:11.849361 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.849341 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:11.852601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.852578 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:09:11.852984 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.852967 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:09:11.861118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.861096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5"] Sep 18 18:09:11.863056 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.863034 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5\" (UID: \"5c5ec621-88bd-4f68-8978-54c1eaf82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:11.963617 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.963576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5\" (UID: \"5c5ec621-88bd-4f68-8978-54c1eaf82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:11.964213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:11.964172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5\" (UID: \"5c5ec621-88bd-4f68-8978-54c1eaf82fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:12.159828 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:12.159795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:12.435012 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:12.434987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5"] Sep 18 18:09:12.437074 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:09:12.437047 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5ec621_88bd_4f68_8978_54c1eaf82fcb.slice/crio-8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e WatchSource:0}: Error finding container 8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e: Status 404 returned error can't find the container with id 8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e Sep 18 18:09:12.630413 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:12.630374 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" event={"ID":"5c5ec621-88bd-4f68-8978-54c1eaf82fcb","Type":"ContainerStarted","Data":"f9e8fa852ad753b8c9be4ae93e1fab911fc37350ccee8390d2470655a85ecad5"} Sep 18 18:09:12.630548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:12.630441 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" event={"ID":"5c5ec621-88bd-4f68-8978-54c1eaf82fcb","Type":"ContainerStarted","Data":"8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e"} Sep 18 18:09:13.637452 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:13.637414 2570 generic.go:356] "Generic (PLEG): container finished" podID="5c5ec621-88bd-4f68-8978-54c1eaf82fcb" containerID="f9e8fa852ad753b8c9be4ae93e1fab911fc37350ccee8390d2470655a85ecad5" exitCode=0 Sep 18 18:09:13.637836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:13.637491 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" event={"ID":"5c5ec621-88bd-4f68-8978-54c1eaf82fcb","Type":"ContainerDied","Data":"f9e8fa852ad753b8c9be4ae93e1fab911fc37350ccee8390d2470655a85ecad5"} Sep 18 18:09:14.777551 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:14.777526 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:14.785296 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:14.785273 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca\") pod \"5c5ec621-88bd-4f68-8978-54c1eaf82fcb\" (UID: \"5c5ec621-88bd-4f68-8978-54c1eaf82fcb\") " Sep 18 18:09:14.785601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:14.785579 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c5ec621-88bd-4f68-8978-54c1eaf82fcb" (UID: "5c5ec621-88bd-4f68-8978-54c1eaf82fcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:09:14.889661 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:14.889622 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ec621-88bd-4f68-8978-54c1eaf82fcb-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:15.651036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:15.651005 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" Sep 18 18:09:15.651231 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:15.651044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5" event={"ID":"5c5ec621-88bd-4f68-8978-54c1eaf82fcb","Type":"ContainerDied","Data":"8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e"} Sep 18 18:09:15.651231 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:15.651095 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f3301c707721864f98d7228139f412712db9f673ff69ec80f614a030f0fbd1e" Sep 18 18:09:23.376622 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.376589 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:09:23.377100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.376870 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" podUID="6d307f29-0800-4630-a81e-9c59a7702f62" containerName="payload-pre-processing" containerID="cri-o://d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94" gracePeriod=30 Sep 18 18:09:23.639644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.639555 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5"] Sep 18 18:09:23.642391 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.642364 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ffa634f-th9f5"] Sep 18 18:09:23.665937 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.665914 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:09:23.711224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.710562 2570 generic.go:356] "Generic (PLEG): container finished" podID="6d307f29-0800-4630-a81e-9c59a7702f62" containerID="d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94" exitCode=0 Sep 18 18:09:23.711224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.710673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" event={"ID":"6d307f29-0800-4630-a81e-9c59a7702f62","Type":"ContainerDied","Data":"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94"} Sep 18 18:09:23.711224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.710726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" event={"ID":"6d307f29-0800-4630-a81e-9c59a7702f62","Type":"ContainerDied","Data":"22ebff3246ff6988c6acd7f51d5809d3b35ddb6d8d707c7555a5a3bb5bf8b4c0"} Sep 18 18:09:23.711224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.710755 2570 scope.go:117] "RemoveContainer" containerID="d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94" Sep 18 18:09:23.711224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.710914 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64" Sep 18 18:09:23.722792 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.722769 2570 scope.go:117] "RemoveContainer" containerID="d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94" Sep 18 18:09:23.723063 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:09:23.723044 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94\": container with ID starting with d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94 not found: ID does not exist" containerID="d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94" Sep 18 18:09:23.723120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.723075 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94"} err="failed to get container status \"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94\": rpc error: code = NotFound desc = could not find container \"d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94\": container with ID starting with d3bd41def6d2651ab1169797f7251bc3e02cdd2a991d6433250948d09325ae94 not found: ID does not exist" Sep 18 18:09:23.769574 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.769544 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume\") pod \"6d307f29-0800-4630-a81e-9c59a7702f62\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " Sep 18 18:09:23.769719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.769586 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2d5\" (UniqueName: \"kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5\") pod \"6d307f29-0800-4630-a81e-9c59a7702f62\" (UID: \"6d307f29-0800-4630-a81e-9c59a7702f62\") " Sep 18 18:09:23.769876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.769854 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d307f29-0800-4630-a81e-9c59a7702f62" (UID: "6d307f29-0800-4630-a81e-9c59a7702f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:09:23.771678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.771654 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5" (OuterVolumeSpecName: "kube-api-access-zs2d5") pod "6d307f29-0800-4630-a81e-9c59a7702f62" (UID: "6d307f29-0800-4630-a81e-9c59a7702f62"). InnerVolumeSpecName "kube-api-access-zs2d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:09:23.871205 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.871157 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307f29-0800-4630-a81e-9c59a7702f62-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:23.871205 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:23.871214 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2d5\" (UniqueName: \"kubernetes.io/projected/6d307f29-0800-4630-a81e-9c59a7702f62-kube-api-access-zs2d5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:24.044646 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:24.044604 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:09:24.050069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:24.050035 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ffa634f-5bb67c58c4-6nf64"] Sep 18 18:09:24.941818 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:24.941783 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c5ec621-88bd-4f68-8978-54c1eaf82fcb" path="/var/lib/kubelet/pods/5c5ec621-88bd-4f68-8978-54c1eaf82fcb/volumes" Sep 18 18:09:24.942214 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:24.942151 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d307f29-0800-4630-a81e-9c59a7702f62" path="/var/lib/kubelet/pods/6d307f29-0800-4630-a81e-9c59a7702f62/volumes" Sep 18 18:09:25.376661 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.376580 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg"] Sep 18 18:09:25.376969 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.376952 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d307f29-0800-4630-a81e-9c59a7702f62" containerName="payload-pre-processing" Sep 18 18:09:25.376969 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.376969 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d307f29-0800-4630-a81e-9c59a7702f62" containerName="payload-pre-processing" Sep 18 18:09:25.377069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.376987 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5ec621-88bd-4f68-8978-54c1eaf82fcb" containerName="revoke-keys" Sep 18 18:09:25.377069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.376993 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5ec621-88bd-4f68-8978-54c1eaf82fcb" containerName="revoke-keys" Sep 18 18:09:25.377069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.377050 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d307f29-0800-4630-a81e-9c59a7702f62" containerName="payload-pre-processing" Sep 18 18:09:25.377069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.377063 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5ec621-88bd-4f68-8978-54c1eaf82fcb" containerName="revoke-keys" Sep 18 18:09:25.381462 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.381445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:25.390070 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.390048 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:09:25.390215 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.390110 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:09:25.400877 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.400853 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg"] Sep 18 18:09:25.485051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.485013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ef5655-prksg\" (UID: \"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:25.586161 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.586126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ef5655-prksg\" (UID: \"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:25.586804 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.586776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ef5655-prksg\" (UID: \"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:25.691343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.691308 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:25.981587 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:25.981563 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg"] Sep 18 18:09:25.983156 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:09:25.983124 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f19c4dd_73a8_4d0e_82af_c5cf2da60b18.slice/crio-59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17 WatchSource:0}: Error finding container 59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17: Status 404 returned error can't find the container with id 59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17 Sep 18 18:09:26.733830 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:26.733801 2570 generic.go:356] "Generic (PLEG): container finished" podID="9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" containerID="2f667297fc86502c20dec609e3198e710b57e6abd34162b60dd6a5ecd2caa321" exitCode=0 Sep 18 18:09:26.733967 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:26.733836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" event={"ID":"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18","Type":"ContainerDied","Data":"2f667297fc86502c20dec609e3198e710b57e6abd34162b60dd6a5ecd2caa321"} Sep 18 18:09:26.733967 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:26.733903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" event={"ID":"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18","Type":"ContainerStarted","Data":"59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17"} Sep 18 18:09:27.873541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:27.873517 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:27.908784 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:27.908752 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca\") pod \"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18\" (UID: \"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18\") " Sep 18 18:09:27.909098 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:27.909073 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" (UID: "9f19c4dd-73a8-4d0e-82af-c5cf2da60b18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:09:28.009896 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:28.009811 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:28.746746 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:28.746716 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" Sep 18 18:09:28.746943 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:28.746749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg" event={"ID":"9f19c4dd-73a8-4d0e-82af-c5cf2da60b18","Type":"ContainerDied","Data":"59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17"} Sep 18 18:09:28.746943 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:28.746800 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ac7308987cffb9b084bc0f3789dbb372310bf9c4078aef0300d1e6d1913a17" Sep 18 18:09:36.570786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.570751 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:09:36.571291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.571017 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" podUID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" containerName="payload-processing" containerID="cri-o://424f2cffa15acc6f79b49ae5b03e4fdfd4c6155521e2baad7ff107f4fc0606aa" gracePeriod=30 Sep 18 18:09:36.576909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.576879 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:09:36.577206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.577141 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" podUID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" containerName="payload-pre-processing" containerID="cri-o://e031882f8b030148426539572eeeff6132f1547222a64cbb3ff5994f6cfa2489" gracePeriod=30 Sep 18 18:09:36.797238 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.797177 2570 generic.go:356] "Generic (PLEG): container finished" podID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" containerID="e031882f8b030148426539572eeeff6132f1547222a64cbb3ff5994f6cfa2489" exitCode=0 Sep 18 18:09:36.797406 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.797345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" event={"ID":"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469","Type":"ContainerDied","Data":"e031882f8b030148426539572eeeff6132f1547222a64cbb3ff5994f6cfa2489"} Sep 18 18:09:36.799486 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.799462 2570 generic.go:356] "Generic (PLEG): container finished" podID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" containerID="424f2cffa15acc6f79b49ae5b03e4fdfd4c6155521e2baad7ff107f4fc0606aa" exitCode=0 Sep 18 18:09:36.799596 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.799532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" event={"ID":"fe8ed9a2-fbc7-4096-8271-cebe2cf36122","Type":"ContainerDied","Data":"424f2cffa15acc6f79b49ae5b03e4fdfd4c6155521e2baad7ff107f4fc0606aa"} Sep 18 18:09:36.876294 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.876271 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:09:36.879526 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.879506 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:09:36.922171 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.922140 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg"] Sep 18 18:09:36.927215 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.927165 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ef5655-prksg"] Sep 18 18:09:36.943280 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.943252 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" path="/var/lib/kubelet/pods/9f19c4dd-73a8-4d0e-82af-c5cf2da60b18/volumes" Sep 18 18:09:36.996514 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.996482 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume\") pod \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " Sep 18 18:09:36.996699 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.996556 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hgk4\" (UniqueName: \"kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4\") pod \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\" (UID: \"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469\") " Sep 18 18:09:36.996699 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.996599 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdcr5\" (UniqueName: \"kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5\") pod \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " Sep 18 18:09:36.996699 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.996629 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume\") pod \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\" (UID: \"fe8ed9a2-fbc7-4096-8271-cebe2cf36122\") " Sep 18 18:09:36.997035 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.997011 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" (UID: "56312c7a-a2b1-4872-a1c5-b8fc1c9fd469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:09:36.997111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.997056 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe8ed9a2-fbc7-4096-8271-cebe2cf36122" (UID: "fe8ed9a2-fbc7-4096-8271-cebe2cf36122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:09:36.998840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.998809 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5" (OuterVolumeSpecName: "kube-api-access-pdcr5") pod "fe8ed9a2-fbc7-4096-8271-cebe2cf36122" (UID: "fe8ed9a2-fbc7-4096-8271-cebe2cf36122"). InnerVolumeSpecName "kube-api-access-pdcr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:09:36.998926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:36.998848 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4" (OuterVolumeSpecName: "kube-api-access-5hgk4") pod "56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" (UID: "56312c7a-a2b1-4872-a1c5-b8fc1c9fd469"). InnerVolumeSpecName "kube-api-access-5hgk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:09:37.098172 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.098133 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:37.098172 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.098164 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hgk4\" (UniqueName: \"kubernetes.io/projected/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469-kube-api-access-5hgk4\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:37.098172 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.098177 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcr5\" (UniqueName: \"kubernetes.io/projected/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-kube-api-access-pdcr5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:37.098414 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.098204 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8ed9a2-fbc7-4096-8271-cebe2cf36122-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:09:37.809349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.809322 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" Sep 18 18:09:37.809788 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.809368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq" event={"ID":"fe8ed9a2-fbc7-4096-8271-cebe2cf36122","Type":"ContainerDied","Data":"522a85fdd85f963091c0d5e2c8d8376e7d59af1b9ae539b17ca8051f6b312998"} Sep 18 18:09:37.809788 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.809439 2570 scope.go:117] "RemoveContainer" containerID="424f2cffa15acc6f79b49ae5b03e4fdfd4c6155521e2baad7ff107f4fc0606aa" Sep 18 18:09:37.810860 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.810840 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" event={"ID":"56312c7a-a2b1-4872-a1c5-b8fc1c9fd469","Type":"ContainerDied","Data":"9e2d0e1e69fc5f75651c6b7b537fd7f1083599e8ef68192ecfa0936d94c7236e"} Sep 18 18:09:37.810945 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.810879 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp" Sep 18 18:09:37.826085 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.826067 2570 scope.go:117] "RemoveContainer" containerID="e031882f8b030148426539572eeeff6132f1547222a64cbb3ff5994f6cfa2489" Sep 18 18:09:37.847586 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.847521 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:09:37.854631 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.854606 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ef5655-6bb97b9c49-kctzq"] Sep 18 18:09:37.879501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.879477 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:09:37.885293 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:37.885271 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ef5655-6f754c4dc4-znrmp"] Sep 18 18:09:38.941762 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:38.941730 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" path="/var/lib/kubelet/pods/56312c7a-a2b1-4872-a1c5-b8fc1c9fd469/volumes" Sep 18 18:09:38.942112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:38.942087 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" path="/var/lib/kubelet/pods/fe8ed9a2-fbc7-4096-8271-cebe2cf36122/volumes" Sep 18 18:09:59.280577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280542 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280871 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" containerName="payload-pre-processing" Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280883 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" containerName="payload-pre-processing" Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280897 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" containerName="payload-processing" Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280903 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" containerName="payload-processing" Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280918 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" containerName="revoke-keys" Sep 18 18:09:59.280932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280923 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" containerName="revoke-keys" Sep 18 18:09:59.281125 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.280990 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8ed9a2-fbc7-4096-8271-cebe2cf36122" containerName="payload-processing" Sep 18 18:09:59.281125 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.281000 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f19c4dd-73a8-4d0e-82af-c5cf2da60b18" containerName="revoke-keys" Sep 18 18:09:59.281125 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.281007 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="56312c7a-a2b1-4872-a1c5-b8fc1c9fd469" containerName="payload-pre-processing" Sep 18 18:09:59.285541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.285525 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.288929 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.288904 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-28ddec\"" Sep 18 18:09:59.289070 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.289012 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-28ddec-dockercfg-4ph6q\"" Sep 18 18:09:59.302904 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.302815 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:09:59.305890 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.305860 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.306023 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.305915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7j5\" (UniqueName: \"kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.307022 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.307000 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:09:59.310158 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.310144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.319291 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.319260 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:09:59.408785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.407492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lxk\" (UniqueName: \"kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.408785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.407559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.408785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.407595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.408785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.407630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7j5\" (UniqueName: \"kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.409141 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.408959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.420752 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.420721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7j5\" (UniqueName: \"kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5\") pod \"payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.514348 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.514316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lxk\" (UniqueName: \"kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.514501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.514367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.515102 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.515084 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.523436 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.523411 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lxk\" (UniqueName: \"kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk\") pod \"payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.595950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.595875 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:09:59.621231 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.621167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:09:59.905538 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.905504 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:09:59.905844 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:09:59.905819 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849ec89f_477f_4a40_99d9_c977a5e08b80.slice/crio-cdd48265f65d093a54e6d7a1154d6ad6869b9750435b1739e487e6843df513ca WatchSource:0}: Error finding container cdd48265f65d093a54e6d7a1154d6ad6869b9750435b1739e487e6843df513ca: Status 404 returned error can't find the container with id cdd48265f65d093a54e6d7a1154d6ad6869b9750435b1739e487e6843df513ca Sep 18 18:09:59.948631 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.948608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:09:59.951284 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:09:59.951261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" event={"ID":"849ec89f-477f-4a40-99d9-c977a5e08b80","Type":"ContainerStarted","Data":"cdd48265f65d093a54e6d7a1154d6ad6869b9750435b1739e487e6843df513ca"} Sep 18 18:09:59.951364 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:09:59.951312 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c677369_69f1_49ae_8a2a_d6f4067153d1.slice/crio-faf44e713e55491b92e0d742ed1760068c506bd71672ef4dad3c5c5c19f6c206 WatchSource:0}: Error finding container faf44e713e55491b92e0d742ed1760068c506bd71672ef4dad3c5c5c19f6c206: Status 404 returned error can't find the container with id faf44e713e55491b92e0d742ed1760068c506bd71672ef4dad3c5c5c19f6c206 Sep 18 18:10:00.957490 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:00.957455 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" event={"ID":"849ec89f-477f-4a40-99d9-c977a5e08b80","Type":"ContainerStarted","Data":"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625"} Sep 18 18:10:00.957947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:00.957522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:10:00.959011 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:00.958986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" event={"ID":"4c677369-69f1-49ae-8a2a-d6f4067153d1","Type":"ContainerStarted","Data":"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8"} Sep 18 18:10:00.959109 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:00.959025 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" event={"ID":"4c677369-69f1-49ae-8a2a-d6f4067153d1","Type":"ContainerStarted","Data":"faf44e713e55491b92e0d742ed1760068c506bd71672ef4dad3c5c5c19f6c206"} Sep 18 18:10:00.959109 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:00.959051 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:10:01.009946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:01.009901 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" podStartSLOduration=2.009888496 podStartE2EDuration="2.009888496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:01.00959843 +0000 UTC m=+1206.102253545" watchObservedRunningTime="2026-09-18 18:10:01.009888496 +0000 UTC m=+1206.102543583" Sep 18 18:10:01.064798 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:01.064747 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" podStartSLOduration=2.064732038 podStartE2EDuration="2.064732038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:01.064415871 +0000 UTC m=+1206.157070957" watchObservedRunningTime="2026-09-18 18:10:01.064732038 +0000 UTC m=+1206.157387192" Sep 18 18:10:09.632629 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.632596 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:10:09.636297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.636276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.638834 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.638810 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"e2e-conflict-a-e4d74f58-kserve-self-signed-certs\"" Sep 18 18:10:09.639605 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.639574 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"kube-root-ca.crt\"" Sep 18 18:10:09.639721 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.639674 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"openshift-service-ca.crt\"" Sep 18 18:10:09.639828 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.639814 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"default-dockercfg-wf7xj\"" Sep 18 18:10:09.651040 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.651015 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:10:09.714270 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.714270 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.714445 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714290 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.714445 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfhf\" (UniqueName: \"kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.714445 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.714548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.714450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816057 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816057 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfhf\" (UniqueName: \"kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816505 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.816887 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.816825 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.819329 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.819280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.820106 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.820080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.829042 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.829020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfhf\" (UniqueName: \"kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf\") pod \"e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:09.945642 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:09.945605 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:10.224254 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:10.224228 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:10:10.225756 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:10:10.225725 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod583120d8_71e3_463f_94a1_f4299fc3eff1.slice/crio-b03247a603c9bb88690eb20d3175575aee35391c43a847f69698d71785f7df2f WatchSource:0}: Error finding container b03247a603c9bb88690eb20d3175575aee35391c43a847f69698d71785f7df2f: Status 404 returned error can't find the container with id b03247a603c9bb88690eb20d3175575aee35391c43a847f69698d71785f7df2f Sep 18 18:10:11.016243 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:11.016168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" event={"ID":"583120d8-71e3-463f-94a1-f4299fc3eff1","Type":"ContainerStarted","Data":"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007"} Sep 18 18:10:11.016243 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:11.016237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" event={"ID":"583120d8-71e3-463f-94a1-f4299fc3eff1","Type":"ContainerStarted","Data":"b03247a603c9bb88690eb20d3175575aee35391c43a847f69698d71785f7df2f"} Sep 18 18:10:11.038270 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:11.038220 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" podStartSLOduration=2.038204378 podStartE2EDuration="2.038204378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:11.035923996 +0000 UTC m=+1216.128579108" watchObservedRunningTime="2026-09-18 18:10:11.038204378 +0000 UTC m=+1216.130859788" Sep 18 18:10:11.967757 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:11.967727 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:10:11.967950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:11.967807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:10:19.739957 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:19.739921 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:10:19.740357 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:19.740144 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" containerID="cri-o://eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f" gracePeriod=30 Sep 18 18:10:19.946112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:19.946072 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:19.946300 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:19.946158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:19.959046 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:19.959021 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:20.111093 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:20.111005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:10:24.997349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:24.997327 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:10:25.107921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.107839 2570 generic.go:356] "Generic (PLEG): container finished" podID="f2055ee3-eedc-41b2-a916-8996011607b6" containerID="eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f" exitCode=0 Sep 18 18:10:25.107921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.107886 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" Sep 18 18:10:25.108117 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.107920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerDied","Data":"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f"} Sep 18 18:10:25.108117 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.107970 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5" event={"ID":"f2055ee3-eedc-41b2-a916-8996011607b6","Type":"ContainerDied","Data":"167ecc67725b90d69d4c86b010ba138351151b8dee372bb52474e69b55dd44ff"} Sep 18 18:10:25.108117 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.108001 2570 scope.go:117] "RemoveContainer" containerID="eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f" Sep 18 18:10:25.117267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.117248 2570 scope.go:117] "RemoveContainer" containerID="c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47" Sep 18 18:10:25.129804 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.126123 2570 scope.go:117] "RemoveContainer" containerID="eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f" Sep 18 18:10:25.132840 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:10:25.132782 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f\": container with ID starting with eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f not found: ID does not exist" containerID="eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f" Sep 18 18:10:25.133069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.133038 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f"} err="failed to get container status \"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f\": rpc error: code = NotFound desc = could not find container \"eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f\": container with ID starting with eda6c763d001bb2be7c8b4498e191bbc98f3535b55f4ac4f852bb77d88da9d8f not found: ID does not exist" Sep 18 18:10:25.133166 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.133073 2570 scope.go:117] "RemoveContainer" containerID="c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47" Sep 18 18:10:25.133685 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:10:25.133658 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47\": container with ID starting with c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47 not found: ID does not exist" containerID="c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47" Sep 18 18:10:25.133767 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.133695 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47"} err="failed to get container status \"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47\": rpc error: code = NotFound desc = could not find container \"c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47\": container with ID starting with c17e7d5b21b39fc04d50ccfbe22da3a44332447bd641f0ce0bd97ddacb21fc47 not found: ID does not exist" Sep 18 18:10:25.162016 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.161986 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162163 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162035 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162163 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162122 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162169 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162223 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162248 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162280 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162516 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162365 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7lpd\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162516 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162403 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket\") pod \"f2055ee3-eedc-41b2-a916-8996011607b6\" (UID: \"f2055ee3-eedc-41b2-a916-8996011607b6\") " Sep 18 18:10:25.162920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162895 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:10:25.163029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.162843 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:10:25.163178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.163132 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:10:25.163178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.163169 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:10:25.163421 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.163366 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:10:25.165662 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.165620 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:10:25.165992 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.165968 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:10:25.166213 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.166167 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:10:25.166533 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.166514 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd" (OuterVolumeSpecName: "kube-api-access-d7lpd") pod "f2055ee3-eedc-41b2-a916-8996011607b6" (UID: "f2055ee3-eedc-41b2-a916-8996011607b6"). InnerVolumeSpecName "kube-api-access-d7lpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:10:25.200063 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.200031 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:10:25.200458 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.200443 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:25.200506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.200462 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:25.200541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.200528 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:25.200575 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.200541 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:25.204836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.204816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.207928 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.207910 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-130a80-openshift-default-dockercfg-2cl5c\"" Sep 18 18:10:25.240700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.240675 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267105 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7lpd\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-kube-api-access-d7lpd\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267141 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267160 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2055ee3-eedc-41b2-a916-8996011607b6-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267174 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267214 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2055ee3-eedc-41b2-a916-8996011607b6-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267227 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267239 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267254 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2055ee3-eedc-41b2-a916-8996011607b6-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.267311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.267274 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2055ee3-eedc-41b2-a916-8996011607b6-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:10:25.368655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368655 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7v7\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.368988 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.368977 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.435333 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.435306 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:10:25.437542 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.437522 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ef5655-openshift-default-f9dd55b6d-ncpz5"] Sep 18 18:10:25.469678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.469794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.469794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.469794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.469794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.469794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7v7\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.469892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.470144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.470166 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.470294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470336 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.470302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.470568 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.470500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.472216 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.472179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.472346 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.472329 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.479004 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.478977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.479420 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.479403 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7v7\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7\") pod \"e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.514257 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.514230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:25.802397 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.802369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:10:25.804286 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:10:25.804257 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b6d5d2_9894_43ab_82c7_0a92c739c1c4.slice/crio-d9977a9053f2dedfe26ca220a282a2c22590d1dad915a1a482beedbbab229234 WatchSource:0}: Error finding container d9977a9053f2dedfe26ca220a282a2c22590d1dad915a1a482beedbbab229234: Status 404 returned error can't find the container with id d9977a9053f2dedfe26ca220a282a2c22590d1dad915a1a482beedbbab229234 Sep 18 18:10:25.806595 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.806561 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:10:25.806692 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.806627 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:10:25.806692 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:25.806663 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:10:26.115971 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.115875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerStarted","Data":"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7"} Sep 18 18:10:26.115971 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.115942 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerStarted","Data":"d9977a9053f2dedfe26ca220a282a2c22590d1dad915a1a482beedbbab229234"} Sep 18 18:10:26.178349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.178301 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podStartSLOduration=1.178285125 podStartE2EDuration="1.178285125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:26.17809837 +0000 UTC m=+1231.270753478" watchObservedRunningTime="2026-09-18 18:10:26.178285125 +0000 UTC m=+1231.270940212" Sep 18 18:10:26.515158 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.515114 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:26.520661 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.520637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:26.942654 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:26.942620 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" path="/var/lib/kubelet/pods/f2055ee3-eedc-41b2-a916-8996011607b6/volumes" Sep 18 18:10:27.122208 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:27.122167 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:27.123028 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:27.123009 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:10:33.399620 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.399577 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:10:33.399977 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.399966 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:33.400030 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.399978 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2055ee3-eedc-41b2-a916-8996011607b6" containerName="istio-proxy" Sep 18 18:10:33.403741 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.403723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.406622 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.406597 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-130a80-dockercfg-hxkzv\"" Sep 18 18:10:33.406622 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.406612 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-130a80\"" Sep 18 18:10:33.417813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.417786 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:10:33.433018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.432985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.433127 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.433110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbwpn\" (UniqueName: \"kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.525473 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.524676 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:10:33.535214 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.534481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwpn\" (UniqueName: \"kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.535214 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.534598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.536282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.535557 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.536282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.535795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.542954 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.542919 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:10:33.551081 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.550217 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwpn\" (UniqueName: \"kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn\") pod \"payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.635365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.635324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vcr\" (UniqueName: \"kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.635499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.635396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.722386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.722357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:33.736919 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.736889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vcr\" (UniqueName: \"kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.737041 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.736961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.737833 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.737810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.745507 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.745487 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vcr\" (UniqueName: \"kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr\") pod \"payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.870991 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.870960 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:33.997509 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:33.997486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:10:33.998578 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:10:33.998544 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcb135a_77cb_42be_8e9b_744930256082.slice/crio-9f9fa3f3fda2093b077dabe97d109640e87528ffa226eb198f8773b468c149d6 WatchSource:0}: Error finding container 9f9fa3f3fda2093b077dabe97d109640e87528ffa226eb198f8773b468c149d6: Status 404 returned error can't find the container with id 9f9fa3f3fda2093b077dabe97d109640e87528ffa226eb198f8773b468c149d6 Sep 18 18:10:34.143946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.143921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:10:34.146518 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:10:34.146488 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa46635_1848_4b29_bb05_3bda1d446c96.slice/crio-b08e66e0ba87a58a60d72a0120d11558574c555b463b95c72df2ca8ac36ede83 WatchSource:0}: Error finding container b08e66e0ba87a58a60d72a0120d11558574c555b463b95c72df2ca8ac36ede83: Status 404 returned error can't find the container with id b08e66e0ba87a58a60d72a0120d11558574c555b463b95c72df2ca8ac36ede83 Sep 18 18:10:34.172763 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.172737 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" event={"ID":"4fa46635-1848-4b29-bb05-3bda1d446c96","Type":"ContainerStarted","Data":"b08e66e0ba87a58a60d72a0120d11558574c555b463b95c72df2ca8ac36ede83"} Sep 18 18:10:34.174358 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.174337 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" event={"ID":"0bcb135a-77cb-42be-8e9b-744930256082","Type":"ContainerStarted","Data":"67702330a681aab975a995f40b1cc916f5357bde974c5dc8f178666e4aa6f3ec"} Sep 18 18:10:34.174451 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.174376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" event={"ID":"0bcb135a-77cb-42be-8e9b-744930256082","Type":"ContainerStarted","Data":"9f9fa3f3fda2093b077dabe97d109640e87528ffa226eb198f8773b468c149d6"} Sep 18 18:10:34.174505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.174463 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:34.196628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:34.196589 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" podStartSLOduration=1.196575126 podStartE2EDuration="1.196575126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:34.194505803 +0000 UTC m=+1239.287160909" watchObservedRunningTime="2026-09-18 18:10:34.196575126 +0000 UTC m=+1239.289230248" Sep 18 18:10:35.181052 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:35.181019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" event={"ID":"4fa46635-1848-4b29-bb05-3bda1d446c96","Type":"ContainerStarted","Data":"3ec16b64af4320c5a22c6b3f2d45d7d608e640b1b6058dbe157e578c45c20c2a"} Sep 18 18:10:35.181507 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:35.181345 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:35.229827 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:35.229784 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" podStartSLOduration=2.229768967 podStartE2EDuration="2.229768967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:35.224258287 +0000 UTC m=+1240.316913390" watchObservedRunningTime="2026-09-18 18:10:35.229768967 +0000 UTC m=+1240.322424054" Sep 18 18:10:45.183219 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:45.183124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:10:46.189844 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:46.189808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:10:57.046155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:57.046124 2570 scope.go:117] "RemoveContainer" containerID="22183f632f4a6917f9d0fc258b13cddcc67330d00e0196ad4406e25ab55f81f4" Sep 18 18:10:57.054573 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:10:57.054556 2570 scope.go:117] "RemoveContainer" containerID="493d2ad89f6b14a0f86043d0d6f773cb4b5fad62a183ba26c9ec1b6c41245b77" Sep 18 18:11:01.347612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.347514 2570 generic.go:356] "Generic (PLEG): container finished" podID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerID="e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7" exitCode=0 Sep 18 18:11:01.347612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.347598 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerDied","Data":"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7"} Sep 18 18:11:01.348129 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.348112 2570 scope.go:117] "RemoveContainer" containerID="e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7" Sep 18 18:11:01.349252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.349227 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:01.349310 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.349292 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:01.349346 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:01.349322 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:02.355257 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:02.355217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerStarted","Data":"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd"} Sep 18 18:11:02.514626 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:02.514591 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:11:02.514884 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:02.514860 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 18 18:11:02.514940 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:02.514920 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 18 18:11:03.514711 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:03.514679 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 18 18:11:03.515097 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:03.514733 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 18 18:11:04.514841 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:04.514802 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 18 18:11:04.515332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:04.514878 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 18 18:11:05.515277 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:05.515238 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 18 18:11:05.515659 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:05.515299 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 18 18:11:06.514687 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:06.514647 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 18 18:11:06.514866 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:06.514713 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 18 18:11:07.543727 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.543698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:11:07.544211 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.544109 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:11:07.544877 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.544848 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:11:07.639899 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.639861 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:11:07.642458 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.642435 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.645038 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.645017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7e72d9f-openshift-default-dockercfg-p2rbx\"" Sep 18 18:11:07.661594 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.661570 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:11:07.673850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.673822 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.673954 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.673853 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.673954 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.673873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.673975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mbj\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.674020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674164 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.674086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674164 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.674123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674164 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.674159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.674335 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.674259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.775104 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.775058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.776435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.776410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.777018 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.776992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.777637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.777618 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778079 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778243 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778431 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mbj\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778594 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778731 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778877 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.778776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.778979 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.776893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.779286 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.777983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.779374 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.776329 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.781786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.779553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.782435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.782414 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.789057 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.789034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.791120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.791095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mbj\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.794612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.794562 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token\") pod \"e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:07.952986 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:07.952957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:08.246159 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.246134 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:11:08.247245 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:11:08.247221 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6ac893_e82b_47cf_83f0_1834c541ea6e.slice/crio-5f68739ebd64e95c64951077d56d730ec7357edd1074f44a70a43aca593474af WatchSource:0}: Error finding container 5f68739ebd64e95c64951077d56d730ec7357edd1074f44a70a43aca593474af: Status 404 returned error can't find the container with id 5f68739ebd64e95c64951077d56d730ec7357edd1074f44a70a43aca593474af Sep 18 18:11:08.249512 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.249479 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:08.249623 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.249553 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:08.249623 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.249596 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:08.398116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.398080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerStarted","Data":"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7"} Sep 18 18:11:08.398328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.398142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerStarted","Data":"5f68739ebd64e95c64951077d56d730ec7357edd1074f44a70a43aca593474af"} Sep 18 18:11:08.424250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.424197 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podStartSLOduration=1.424155544 podStartE2EDuration="1.424155544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:08.420420734 +0000 UTC m=+1273.513075846" watchObservedRunningTime="2026-09-18 18:11:08.424155544 +0000 UTC m=+1273.516810629" Sep 18 18:11:08.954071 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:08.954040 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:09.958946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:09.958916 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:10.412930 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:10.412895 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:10.413905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:10.413882 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:41.137568 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.137534 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58"] Sep 18 18:11:41.140436 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.140420 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:41.143383 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.143360 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:11:41.143761 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.143745 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:11:41.151141 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.151119 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58"] Sep 18 18:11:41.306262 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.306228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-502939-zhr58\" (UID: \"7e91e176-7085-479c-bac7-0c6be481586c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:41.407636 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.407606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-502939-zhr58\" (UID: \"7e91e176-7085-479c-bac7-0c6be481586c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:41.408422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.408404 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-502939-zhr58\" (UID: \"7e91e176-7085-479c-bac7-0c6be481586c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:41.452067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.452032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:41.453372 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.453337 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 18:11:41.453487 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.453421 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 18:11:41.728360 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:41.728332 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58"] Sep 18 18:11:41.731644 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:11:41.731613 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e91e176_7085_479c_bac7_0c6be481586c.slice/crio-e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778 WatchSource:0}: Error finding container e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778: Status 404 returned error can't find the container with id e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778 Sep 18 18:11:42.606112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.606028 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e91e176-7085-479c-bac7-0c6be481586c" containerID="0dc12d08542b64b330d4cea0166cef7195bcf25ef609dad87f95cf9b1c043ac5" exitCode=0 Sep 18 18:11:42.606596 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.606110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" event={"ID":"7e91e176-7085-479c-bac7-0c6be481586c","Type":"ContainerDied","Data":"0dc12d08542b64b330d4cea0166cef7195bcf25ef609dad87f95cf9b1c043ac5"} Sep 18 18:11:42.606596 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.606164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" event={"ID":"7e91e176-7085-479c-bac7-0c6be481586c","Type":"ContainerStarted","Data":"e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778"} Sep 18 18:11:42.607781 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.607762 2570 generic.go:356] "Generic (PLEG): container finished" podID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerID="1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7" exitCode=0 Sep 18 18:11:42.607850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.607818 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerDied","Data":"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7"} Sep 18 18:11:42.608176 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.608164 2570 scope.go:117] "RemoveContainer" containerID="1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7" Sep 18 18:11:42.609404 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.609380 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:42.609471 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.609446 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:42.609508 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:42.609477 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:11:43.617563 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.617524 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerStarted","Data":"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b"} Sep 18 18:11:43.765736 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.765706 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:43.832142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.832103 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca\") pod \"7e91e176-7085-479c-bac7-0c6be481586c\" (UID: \"7e91e176-7085-479c-bac7-0c6be481586c\") " Sep 18 18:11:43.832454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.832430 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e91e176-7085-479c-bac7-0c6be481586c" (UID: "7e91e176-7085-479c-bac7-0c6be481586c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:11:43.832523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.832514 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e176-7085-479c-bac7-0c6be481586c-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:11:43.953503 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.953470 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:43.953688 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.953662 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 18 18:11:43.953742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:43.953721 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 18 18:11:44.624847 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:44.624819 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" Sep 18 18:11:44.625261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:44.624853 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58" event={"ID":"7e91e176-7085-479c-bac7-0c6be481586c","Type":"ContainerDied","Data":"e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778"} Sep 18 18:11:44.625261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:44.624902 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8341ecd42c93c6bf7c523a27f059063798bb6ab9a152d4adcc3d2aee69e2778" Sep 18 18:11:44.954309 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:44.954273 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 18 18:11:44.954467 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:44.954335 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 18 18:11:45.953605 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:45.953563 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 18 18:11:45.954007 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:45.953623 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 18 18:11:46.953831 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:46.953799 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 18 18:11:46.954246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:46.953850 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 18 18:11:47.953966 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:47.953927 2570 patch_prober.go:28] interesting pod/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 18 18:11:47.954382 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:47.953996 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 18 18:11:48.957504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:48.957471 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:48.957947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:48.957760 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:48.958548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:48.958532 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:11:51.923300 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:51.923264 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58"] Sep 18 18:11:51.925713 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:51.925691 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-502939-zhr58"] Sep 18 18:11:52.942406 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:52.942373 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e91e176-7085-479c-bac7-0c6be481586c" path="/var/lib/kubelet/pods/7e91e176-7085-479c-bac7-0c6be481586c/volumes" Sep 18 18:11:57.082044 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:11:57.082009 2570 scope.go:117] "RemoveContainer" containerID="e8239cc5d38bb7053a3845b75edfdbb18c5599bd79a1f79d2825af69c18b2d9d" Sep 18 18:12:34.732425 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:34.732390 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:12:34.732875 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:34.732615 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" containerID="cri-o://62340a5c352b3c898dbcf6e059c3744f134163e3877bdb51e9f19e638aeee0f2" gracePeriod=30 Sep 18 18:12:35.083585 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.083545 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9"] Sep 18 18:12:35.083979 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.083966 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e91e176-7085-479c-bac7-0c6be481586c" containerName="revoke-keys" Sep 18 18:12:35.084033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.083980 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e91e176-7085-479c-bac7-0c6be481586c" containerName="revoke-keys" Sep 18 18:12:35.084082 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.084038 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e91e176-7085-479c-bac7-0c6be481586c" containerName="revoke-keys" Sep 18 18:12:35.086196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.086166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:35.089122 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.089100 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:12:35.089223 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.089127 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:12:35.097032 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.097002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9"] Sep 18 18:12:35.139816 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.139769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9\" (UID: \"311da3a5-94ca-42b1-8070-5822a64b1ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:35.240732 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.240689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9\" (UID: \"311da3a5-94ca-42b1-8070-5822a64b1ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:35.241441 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.241418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9\" (UID: \"311da3a5-94ca-42b1-8070-5822a64b1ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:35.398632 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.398527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:35.675959 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.675929 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9"] Sep 18 18:12:35.678220 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:12:35.678166 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311da3a5_94ca_42b1_8070_5822a64b1ba9.slice/crio-4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2 WatchSource:0}: Error finding container 4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2: Status 404 returned error can't find the container with id 4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2 Sep 18 18:12:35.922407 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.922376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" event={"ID":"311da3a5-94ca-42b1-8070-5822a64b1ba9","Type":"ContainerStarted","Data":"f5de5c4766a2fa549311ee34b068078f65a88df0a69249c7d8fafab7d5b1d076"} Sep 18 18:12:35.922771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.922425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" event={"ID":"311da3a5-94ca-42b1-8070-5822a64b1ba9","Type":"ContainerStarted","Data":"4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2"} Sep 18 18:12:35.940385 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:35.940290 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" podStartSLOduration=0.940275972 podStartE2EDuration="940.275972ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:12:35.93857752 +0000 UTC m=+1361.031232638" watchObservedRunningTime="2026-09-18 18:12:35.940275972 +0000 UTC m=+1361.032931059" Sep 18 18:12:36.929946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:36.929908 2570 generic.go:356] "Generic (PLEG): container finished" podID="311da3a5-94ca-42b1-8070-5822a64b1ba9" containerID="f5de5c4766a2fa549311ee34b068078f65a88df0a69249c7d8fafab7d5b1d076" exitCode=0 Sep 18 18:12:36.930351 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:36.929990 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" event={"ID":"311da3a5-94ca-42b1-8070-5822a64b1ba9","Type":"ContainerDied","Data":"f5de5c4766a2fa549311ee34b068078f65a88df0a69249c7d8fafab7d5b1d076"} Sep 18 18:12:38.076285 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.076260 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:38.171133 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.171098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca\") pod \"311da3a5-94ca-42b1-8070-5822a64b1ba9\" (UID: \"311da3a5-94ca-42b1-8070-5822a64b1ba9\") " Sep 18 18:12:38.171528 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.171505 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "311da3a5-94ca-42b1-8070-5822a64b1ba9" (UID: "311da3a5-94ca-42b1-8070-5822a64b1ba9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:12:38.272581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.272493 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311da3a5-94ca-42b1-8070-5822a64b1ba9-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:38.576297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.576210 2570 patch_prober.go:28] interesting pod/e2e-inf-a-502939-openshift-default-545646f854-vnvkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:12:38.576297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.576270 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:12:38.943672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.943631 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" event={"ID":"311da3a5-94ca-42b1-8070-5822a64b1ba9","Type":"ContainerDied","Data":"4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2"} Sep 18 18:12:38.943880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.943688 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a607611ae049d7ffa57dca8f9866923a6f55798f7d3b735fb2bdb2b4fc291c2" Sep 18 18:12:38.943880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:38.943651 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9" Sep 18 18:12:39.953934 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:39.953896 2570 generic.go:356] "Generic (PLEG): container finished" podID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerID="62340a5c352b3c898dbcf6e059c3744f134163e3877bdb51e9f19e638aeee0f2" exitCode=0 Sep 18 18:12:39.954433 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:39.953970 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerDied","Data":"62340a5c352b3c898dbcf6e059c3744f134163e3877bdb51e9f19e638aeee0f2"} Sep 18 18:12:39.954433 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:39.954082 2570 scope.go:117] "RemoveContainer" containerID="1f0f37e8d8a306af148dc719b9cf79503ec988086430228f60632ec4b851a3a2" Sep 18 18:12:39.987547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:39.987519 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:12:40.091901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.091807 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.091917 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.091946 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.091986 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092019 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4q4t\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092102 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092117 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092149 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092179 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092218 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:12:40.092330 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092268 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert\") pod \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\" (UID: \"aeb5b098-d016-4c37-88b3-1d6864dff6f5\") " Sep 18 18:12:40.092637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092395 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:12:40.092637 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092482 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:12:40.092751 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092712 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:12:40.092751 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092732 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.092836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092753 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.092836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092766 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.092836 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.092779 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.094266 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.094241 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:12:40.094389 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.094302 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:12:40.094615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.094592 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:12:40.094615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.094600 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t" (OuterVolumeSpecName: "kube-api-access-t4q4t") pod "aeb5b098-d016-4c37-88b3-1d6864dff6f5" (UID: "aeb5b098-d016-4c37-88b3-1d6864dff6f5"). InnerVolumeSpecName "kube-api-access-t4q4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:12:40.194067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.194031 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.194067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.194061 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.194067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.194070 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.194067 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.194077 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4q4t\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-kube-api-access-t4q4t\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.194365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.194086 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb5b098-d016-4c37-88b3-1d6864dff6f5-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:40.960913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.960885 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" Sep 18 18:12:40.961344 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.960912 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh" event={"ID":"aeb5b098-d016-4c37-88b3-1d6864dff6f5","Type":"ContainerDied","Data":"5c01fc1aa6f1fb66805cece93c57b44b4b90931ef5269fb604e6bf45a58a3104"} Sep 18 18:12:40.961344 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.960969 2570 scope.go:117] "RemoveContainer" containerID="62340a5c352b3c898dbcf6e059c3744f134163e3877bdb51e9f19e638aeee0f2" Sep 18 18:12:40.981686 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.981658 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:12:40.987532 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:40.987509 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-502939-openshift-default-545646f854-vnvkh"] Sep 18 18:12:42.942259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:42.942227 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" path="/var/lib/kubelet/pods/aeb5b098-d016-4c37-88b3-1d6864dff6f5/volumes" Sep 18 18:12:45.668622 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:45.668585 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:12:45.669032 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:45.668854 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" podUID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" containerName="payload-pre-processing" containerID="cri-o://2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee" gracePeriod=30 Sep 18 18:12:45.864093 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:45.864054 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9"] Sep 18 18:12:45.869238 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:45.867469 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ee3049-p7dr9"] Sep 18 18:12:45.944602 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:45.944577 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:12:46.016822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.016158 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" containerID="2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee" exitCode=0 Sep 18 18:12:46.016822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.016272 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" event={"ID":"1a9cfb68-50b7-493a-af16-7f2e3c4aa254","Type":"ContainerDied","Data":"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee"} Sep 18 18:12:46.016822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.016324 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" event={"ID":"1a9cfb68-50b7-493a-af16-7f2e3c4aa254","Type":"ContainerDied","Data":"279ffa64ad53dd9403c7c8717fc32be16708e77d704b7c4185e0cf81a3642d16"} Sep 18 18:12:46.016822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.016356 2570 scope.go:117] "RemoveContainer" containerID="2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee" Sep 18 18:12:46.016822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.016500 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk" Sep 18 18:12:46.041517 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.041483 2570 scope.go:117] "RemoveContainer" containerID="2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee" Sep 18 18:12:46.041991 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:12:46.041962 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee\": container with ID starting with 2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee not found: ID does not exist" containerID="2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee" Sep 18 18:12:46.042108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.042005 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee"} err="failed to get container status \"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee\": rpc error: code = NotFound desc = could not find container \"2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee\": container with ID starting with 2223e41df0e0a7e7cc057fea0c1627e2bd026247399d7913515b1929214b8cee not found: ID does not exist" Sep 18 18:12:46.052585 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.052559 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svtzf\" (UniqueName: \"kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf\") pod \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " Sep 18 18:12:46.052698 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.052615 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume\") pod \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\" (UID: \"1a9cfb68-50b7-493a-af16-7f2e3c4aa254\") " Sep 18 18:12:46.052973 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.052945 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a9cfb68-50b7-493a-af16-7f2e3c4aa254" (UID: "1a9cfb68-50b7-493a-af16-7f2e3c4aa254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:12:46.054639 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.054609 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf" (OuterVolumeSpecName: "kube-api-access-svtzf") pod "1a9cfb68-50b7-493a-af16-7f2e3c4aa254" (UID: "1a9cfb68-50b7-493a-af16-7f2e3c4aa254"). InnerVolumeSpecName "kube-api-access-svtzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:12:46.153908 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.153875 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtzf\" (UniqueName: \"kubernetes.io/projected/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-kube-api-access-svtzf\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:46.153908 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.153907 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9cfb68-50b7-493a-af16-7f2e3c4aa254-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:12:46.352176 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.352141 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:12:46.356454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.356427 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee3049-66bc74df8b-r2nwk"] Sep 18 18:12:46.941862 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.941830 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" path="/var/lib/kubelet/pods/1a9cfb68-50b7-493a-af16-7f2e3c4aa254/volumes" Sep 18 18:12:46.942234 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:46.942204 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311da3a5-94ca-42b1-8070-5822a64b1ba9" path="/var/lib/kubelet/pods/311da3a5-94ca-42b1-8070-5822a64b1ba9/volumes" Sep 18 18:12:57.104060 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:57.104030 2570 scope.go:117] "RemoveContainer" containerID="42b33c9351da3b2cf3d93f2a87f0e4c935fdc8b7ed69242cbfe833321f28142f" Sep 18 18:12:57.112827 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:12:57.112806 2570 scope.go:117] "RemoveContainer" containerID="5ae51465463797ece3d673c756acd975b84d6327c81c738b5fd2a28ff4104a00" Sep 18 18:13:20.003753 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.003719 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh"] Sep 18 18:13:20.004288 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004268 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.004342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004292 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.004342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004307 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" containerName="payload-pre-processing" Sep 18 18:13:20.004342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004316 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" containerName="payload-pre-processing" Sep 18 18:13:20.004342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004330 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.004342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004339 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004358 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311da3a5-94ca-42b1-8070-5822a64b1ba9" containerName="revoke-keys" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004369 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="311da3a5-94ca-42b1-8070-5822a64b1ba9" containerName="revoke-keys" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004459 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004470 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9cfb68-50b7-493a-af16-7f2e3c4aa254" containerName="payload-pre-processing" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004481 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="311da3a5-94ca-42b1-8070-5822a64b1ba9" containerName="revoke-keys" Sep 18 18:13:20.004501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.004493 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb5b098-d016-4c37-88b3-1d6864dff6f5" containerName="istio-proxy" Sep 18 18:13:20.006577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.006562 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:20.009239 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.009216 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:13:20.009368 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.009220 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:13:20.016386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.016363 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh"] Sep 18 18:13:20.078002 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.077964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh\" (UID: \"3bf57129-3a3c-4ff3-9300-2175884f5775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:20.178952 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.178919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh\" (UID: \"3bf57129-3a3c-4ff3-9300-2175884f5775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:20.179617 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.179598 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh\" (UID: \"3bf57129-3a3c-4ff3-9300-2175884f5775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:20.317742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.317657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:20.615385 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:20.615359 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh"] Sep 18 18:13:20.616268 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:13:20.616242 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf57129_3a3c_4ff3_9300_2175884f5775.slice/crio-0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7 WatchSource:0}: Error finding container 0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7: Status 404 returned error can't find the container with id 0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7 Sep 18 18:13:21.224633 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:21.224597 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" event={"ID":"3bf57129-3a3c-4ff3-9300-2175884f5775","Type":"ContainerStarted","Data":"1a7900885621c306913a1424935a369a4d58b49b4fe3b8d2e4c26210572a0b81"} Sep 18 18:13:21.224983 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:21.224650 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" event={"ID":"3bf57129-3a3c-4ff3-9300-2175884f5775","Type":"ContainerStarted","Data":"0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7"} Sep 18 18:13:21.245870 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:21.245821 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" podStartSLOduration=2.245807319 podStartE2EDuration="2.245807319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:13:21.243267697 +0000 UTC m=+1406.335922800" watchObservedRunningTime="2026-09-18 18:13:21.245807319 +0000 UTC m=+1406.338462406" Sep 18 18:13:22.232814 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:22.232778 2570 generic.go:356] "Generic (PLEG): container finished" podID="3bf57129-3a3c-4ff3-9300-2175884f5775" containerID="1a7900885621c306913a1424935a369a4d58b49b4fe3b8d2e4c26210572a0b81" exitCode=0 Sep 18 18:13:22.233224 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:22.232875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" event={"ID":"3bf57129-3a3c-4ff3-9300-2175884f5775","Type":"ContainerDied","Data":"1a7900885621c306913a1424935a369a4d58b49b4fe3b8d2e4c26210572a0b81"} Sep 18 18:13:23.374232 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:23.374206 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:23.509711 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:23.509619 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca\") pod \"3bf57129-3a3c-4ff3-9300-2175884f5775\" (UID: \"3bf57129-3a3c-4ff3-9300-2175884f5775\") " Sep 18 18:13:23.509966 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:23.509944 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bf57129-3a3c-4ff3-9300-2175884f5775" (UID: "3bf57129-3a3c-4ff3-9300-2175884f5775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:13:23.610740 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:23.610704 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57129-3a3c-4ff3-9300-2175884f5775-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:24.246247 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:24.246219 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" Sep 18 18:13:24.246247 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:24.246219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh" event={"ID":"3bf57129-3a3c-4ff3-9300-2175884f5775","Type":"ContainerDied","Data":"0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7"} Sep 18 18:13:24.246489 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:24.246281 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4cbb6b8efe20d13901060978fb75b44c4fdaf5e348533022c6ace1d88d21e7" Sep 18 18:13:28.961289 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:28.961251 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:13:28.961759 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:28.961493 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" containerID="cri-o://22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a" gracePeriod=30 Sep 18 18:13:31.732005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:31.731968 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:13:31.732447 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:31.732200 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" podUID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" containerName="payload-pre-processing" containerID="cri-o://d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312" gracePeriod=30 Sep 18 18:13:31.989438 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:31.989382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:13:32.006562 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.006537 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh"] Sep 18 18:13:32.014218 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.014177 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-609aba-plmdh"] Sep 18 18:13:32.095568 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.095537 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume\") pod \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " Sep 18 18:13:32.095720 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.095580 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w28g\" (UniqueName: \"kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g\") pod \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\" (UID: \"d9bd9803-e7b5-4d21-8660-d0c1000e555a\") " Sep 18 18:13:32.095994 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.095955 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9bd9803-e7b5-4d21-8660-d0c1000e555a" (UID: "d9bd9803-e7b5-4d21-8660-d0c1000e555a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:13:32.097778 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.097758 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g" (OuterVolumeSpecName: "kube-api-access-8w28g") pod "d9bd9803-e7b5-4d21-8660-d0c1000e555a" (UID: "d9bd9803-e7b5-4d21-8660-d0c1000e555a"). InnerVolumeSpecName "kube-api-access-8w28g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:13:32.197676 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.197615 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bd9803-e7b5-4d21-8660-d0c1000e555a-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:32.197676 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.197650 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w28g\" (UniqueName: \"kubernetes.io/projected/d9bd9803-e7b5-4d21-8660-d0c1000e555a-kube-api-access-8w28g\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:32.295290 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.295193 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" containerID="d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312" exitCode=0 Sep 18 18:13:32.295290 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.295244 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" Sep 18 18:13:32.295506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.295284 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" event={"ID":"d9bd9803-e7b5-4d21-8660-d0c1000e555a","Type":"ContainerDied","Data":"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312"} Sep 18 18:13:32.295506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.295348 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8" event={"ID":"d9bd9803-e7b5-4d21-8660-d0c1000e555a","Type":"ContainerDied","Data":"d468f53e6072ab01b979b237ee4e4289fa0c70bf328a71faf6cd040041a5dc51"} Sep 18 18:13:32.295506 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.295370 2570 scope.go:117] "RemoveContainer" containerID="d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312" Sep 18 18:13:32.305652 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.305633 2570 scope.go:117] "RemoveContainer" containerID="d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312" Sep 18 18:13:32.305928 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:13:32.305909 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312\": container with ID starting with d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312 not found: ID does not exist" containerID="d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312" Sep 18 18:13:32.305998 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.305938 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312"} err="failed to get container status \"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312\": rpc error: code = NotFound desc = could not find container \"d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312\": container with ID starting with d9311316521e9b4b07f84e67ec79561e5bdc1096d7206594d37608b765a23312 not found: ID does not exist" Sep 18 18:13:32.317742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.317703 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:13:32.324880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.324856 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-609aba-5b5c77dc6b-fjmw8"] Sep 18 18:13:32.942985 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.942950 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf57129-3a3c-4ff3-9300-2175884f5775" path="/var/lib/kubelet/pods/3bf57129-3a3c-4ff3-9300-2175884f5775/volumes" Sep 18 18:13:32.943401 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:32.943352 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" path="/var/lib/kubelet/pods/d9bd9803-e7b5-4d21-8660-d0c1000e555a/volumes" Sep 18 18:13:33.622945 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:33.622906 2570 patch_prober.go:28] interesting pod/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:13:33.623114 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:33.622977 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:13:34.218409 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.218385 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:13:34.308982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.308896 2570 generic.go:356] "Generic (PLEG): container finished" podID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerID="22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a" exitCode=0 Sep 18 18:13:34.308982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.308946 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" Sep 18 18:13:34.309196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.308987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerDied","Data":"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a"} Sep 18 18:13:34.309196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.309047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw" event={"ID":"74fbced2-da3a-48d1-89ca-f02b186d576a","Type":"ContainerDied","Data":"dba4510f98ccf80b0ac8a38cfcce1c91c06e8b52597f2847029142189662804e"} Sep 18 18:13:34.309196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.309070 2570 scope.go:117] "RemoveContainer" containerID="22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a" Sep 18 18:13:34.316643 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316619 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316656 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316693 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316739 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316942 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316791 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqbwp\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316942 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316827 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316942 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316878 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.316942 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316911 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:34.317142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.317142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.316978 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:34.317142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317011 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token\") pod \"74fbced2-da3a-48d1-89ca-f02b186d576a\" (UID: \"74fbced2-da3a-48d1-89ca-f02b186d576a\") " Sep 18 18:13:34.317142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317065 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:34.317392 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317161 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data" (OuterVolumeSpecName: "istio-data") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:34.317392 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317259 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:13:34.317499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317405 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.317499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317425 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.317499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317438 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.317499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317451 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.317499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.317463 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fbced2-da3a-48d1-89ca-f02b186d576a-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.319062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.318953 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:34.319135 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.319116 2570 scope.go:117] "RemoveContainer" containerID="eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd" Sep 18 18:13:34.319219 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.319157 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp" (OuterVolumeSpecName: "kube-api-access-bqbwp") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "kube-api-access-bqbwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:13:34.319275 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.319252 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:13:34.319540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.319517 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token" (OuterVolumeSpecName: "istio-token") pod "74fbced2-da3a-48d1-89ca-f02b186d576a" (UID: "74fbced2-da3a-48d1-89ca-f02b186d576a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:13:34.337326 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.337299 2570 scope.go:117] "RemoveContainer" containerID="22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a" Sep 18 18:13:34.337627 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:13:34.337608 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a\": container with ID starting with 22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a not found: ID does not exist" containerID="22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a" Sep 18 18:13:34.337675 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.337640 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a"} err="failed to get container status \"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a\": rpc error: code = NotFound desc = could not find container \"22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a\": container with ID starting with 22038075bc86b99647250f74bd9aaf90e466ce23600e240606ddc849b209203a not found: ID does not exist" Sep 18 18:13:34.337675 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.337658 2570 scope.go:117] "RemoveContainer" containerID="eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd" Sep 18 18:13:34.337932 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:13:34.337914 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd\": container with ID starting with eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd not found: ID does not exist" containerID="eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd" Sep 18 18:13:34.337990 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.337939 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd"} err="failed to get container status \"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd\": rpc error: code = NotFound desc = could not find container \"eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd\": container with ID starting with eee0bfcc67a2d717e5d0238f71cf8b9a4d7ef00ff5f122f3f7c8e30a37a48edd not found: ID does not exist" Sep 18 18:13:34.418679 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.418647 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqbwp\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-kube-api-access-bqbwp\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.418679 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.418674 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.418841 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.418687 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.418841 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.418699 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fbced2-da3a-48d1-89ca-f02b186d576a-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:34.635490 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.635464 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:13:34.638195 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.638159 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ee3049-openshift-default-7b4b5646b8-wh9bw"] Sep 18 18:13:34.942423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:34.942394 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" path="/var/lib/kubelet/pods/74fbced2-da3a-48d1-89ca-f02b186d576a/volumes" Sep 18 18:13:36.880220 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880173 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880536 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880546 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880558 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" containerName="payload-pre-processing" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880565 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" containerName="payload-pre-processing" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880573 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880578 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880596 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf57129-3a3c-4ff3-9300-2175884f5775" containerName="revoke-keys" Sep 18 18:13:36.880612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880602 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf57129-3a3c-4ff3-9300-2175884f5775" containerName="revoke-keys" Sep 18 18:13:36.880889 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880654 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9bd9803-e7b5-4d21-8660-d0c1000e555a" containerName="payload-pre-processing" Sep 18 18:13:36.880889 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880661 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880889 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880668 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fbced2-da3a-48d1-89ca-f02b186d576a" containerName="istio-proxy" Sep 18 18:13:36.880889 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.880676 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf57129-3a3c-4ff3-9300-2175884f5775" containerName="revoke-keys" Sep 18 18:13:36.883569 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.883551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:36.885951 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.885927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-53cde3\"" Sep 18 18:13:36.896054 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.896032 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:13:36.939888 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.939854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpgl5\" (UniqueName: \"kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:36.940035 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:36.939949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.041631 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.041588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgl5\" (UniqueName: \"kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.041802 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.041714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.042853 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.042822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.056046 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.056021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgl5\" (UniqueName: \"kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5\") pod \"payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.193868 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.193832 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:37.468090 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:37.468065 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:13:37.470061 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:13:37.470035 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3161b5f_3ba6_4686_b18e_6d286e0ff09f.slice/crio-1545ea571d2aad91fdb4ed84d76ffd692ba1c5f2d16e76aeee349342b113f2e4 WatchSource:0}: Error finding container 1545ea571d2aad91fdb4ed84d76ffd692ba1c5f2d16e76aeee349342b113f2e4: Status 404 returned error can't find the container with id 1545ea571d2aad91fdb4ed84d76ffd692ba1c5f2d16e76aeee349342b113f2e4 Sep 18 18:13:38.337742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:38.337704 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" event={"ID":"b3161b5f-3ba6-4686-b18e-6d286e0ff09f","Type":"ContainerStarted","Data":"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74"} Sep 18 18:13:38.338142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:38.337759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" event={"ID":"b3161b5f-3ba6-4686-b18e-6d286e0ff09f","Type":"ContainerStarted","Data":"1545ea571d2aad91fdb4ed84d76ffd692ba1c5f2d16e76aeee349342b113f2e4"} Sep 18 18:13:38.338142 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:38.337802 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:13:38.357461 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:38.357396 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" podStartSLOduration=2.357377285 podStartE2EDuration="2.357377285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:13:38.354632371 +0000 UTC m=+1423.447287472" watchObservedRunningTime="2026-09-18 18:13:38.357377285 +0000 UTC m=+1423.450032413" Sep 18 18:13:39.611501 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:39.611422 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:13:39.611881 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:39.611661 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" containerID="cri-o://22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe" gracePeriod=30 Sep 18 18:13:44.862937 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.862914 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:13:44.916518 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916487 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916526 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916553 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916605 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lslj\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916647 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916892 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916688 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916892 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916725 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.916892 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.916804 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:44.917043 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917010 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:44.917126 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917108 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.917207 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917131 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:44.917284 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917261 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert\") pod \"5773d6a9-a8a4-4d95-8112-6862993adac4\" (UID: \"5773d6a9-a8a4-4d95-8112-6862993adac4\") " Sep 18 18:13:44.917378 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917358 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:44.917661 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917634 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:44.917786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917663 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:44.917786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917678 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:44.917786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917692 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:44.917786 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.917674 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:13:44.918933 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.918906 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:13:44.918933 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.918915 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj" (OuterVolumeSpecName: "kube-api-access-6lslj") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "kube-api-access-6lslj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:13:44.919101 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.918954 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:13:44.919245 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:44.919226 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5773d6a9-a8a4-4d95-8112-6862993adac4" (UID: "5773d6a9-a8a4-4d95-8112-6862993adac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:13:45.018455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.018427 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773d6a9-a8a4-4d95-8112-6862993adac4-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:45.018455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.018453 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:45.018455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.018461 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:45.018644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.018470 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lslj\" (UniqueName: \"kubernetes.io/projected/5773d6a9-a8a4-4d95-8112-6862993adac4-kube-api-access-6lslj\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:45.018644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.018479 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773d6a9-a8a4-4d95-8112-6862993adac4-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:13:45.377808 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.377724 2570 generic.go:356] "Generic (PLEG): container finished" podID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerID="22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe" exitCode=0 Sep 18 18:13:45.377808 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.377778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerDied","Data":"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe"} Sep 18 18:13:45.377808 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.377800 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" Sep 18 18:13:45.378026 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.377880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc" event={"ID":"5773d6a9-a8a4-4d95-8112-6862993adac4","Type":"ContainerDied","Data":"dfa67e263fbfd20db171d36bde707676f47b3ce9ac361750918e6f588d05d2b7"} Sep 18 18:13:45.378026 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.377913 2570 scope.go:117] "RemoveContainer" containerID="22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe" Sep 18 18:13:45.386930 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.386912 2570 scope.go:117] "RemoveContainer" containerID="903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db" Sep 18 18:13:45.395232 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.395214 2570 scope.go:117] "RemoveContainer" containerID="22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe" Sep 18 18:13:45.395538 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:13:45.395519 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe\": container with ID starting with 22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe not found: ID does not exist" containerID="22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe" Sep 18 18:13:45.395601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.395548 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe"} err="failed to get container status \"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe\": rpc error: code = NotFound desc = could not find container \"22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe\": container with ID starting with 22472927a12cfd2c4e7b57da5f098e23c951c102974aa150946d2e37dfe4edfe not found: ID does not exist" Sep 18 18:13:45.395601 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.395565 2570 scope.go:117] "RemoveContainer" containerID="903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db" Sep 18 18:13:45.395873 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:13:45.395846 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db\": container with ID starting with 903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db not found: ID does not exist" containerID="903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db" Sep 18 18:13:45.395929 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.395882 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db"} err="failed to get container status \"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db\": rpc error: code = NotFound desc = could not find container \"903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db\": container with ID starting with 903d3c79fd548c326b3f9bf96019175745bcbf286b26dd877c32d190d96781db not found: ID does not exist" Sep 18 18:13:45.396363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.396343 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:13:45.402353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:45.402332 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-609aba-openshift-default-58668d4c86-slhpc"] Sep 18 18:13:46.942495 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:46.942465 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" path="/var/lib/kubelet/pods/5773d6a9-a8a4-4d95-8112-6862993adac4/volumes" Sep 18 18:13:49.344753 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:13:49.344724 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:14:11.310658 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.310613 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:14:11.311167 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311132 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.311167 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311153 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.311341 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311202 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.311341 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311212 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.311341 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311310 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.311341 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.311328 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5773d6a9-a8a4-4d95-8112-6862993adac4" containerName="istio-proxy" Sep 18 18:14:11.316265 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.316243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.318874 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.318845 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-341bd4\"" Sep 18 18:14:11.318983 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.318872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-341bd4-dockercfg-6khlk\"" Sep 18 18:14:11.325094 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.325068 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:14:11.329024 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.329005 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:14:11.329114 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.329083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.341591 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.341567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:14:11.378665 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.378629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.378665 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.378670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.378886 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.378706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dmf\" (UniqueName: \"kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.378886 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.378767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvpb\" (UniqueName: \"kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.479546 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.479509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvpb\" (UniqueName: \"kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.479726 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.479611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.479726 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.479644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.479726 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.479696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dmf\" (UniqueName: \"kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.480406 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.480387 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.480451 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.480415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.487845 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.487816 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvpb\" (UniqueName: \"kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb\") pod \"payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.487996 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.487979 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dmf\" (UniqueName: \"kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf\") pod \"payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.628840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.628740 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:11.641002 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.640966 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:11.906921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.906893 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:14:11.908738 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:14:11.908704 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816e406b_7b02_4bf3_b0f7_be41f02c88e5.slice/crio-e5b24c3438cc401dd50211a6898bf330feab14f77d3a3557d83bf7b30aee8c93 WatchSource:0}: Error finding container e5b24c3438cc401dd50211a6898bf330feab14f77d3a3557d83bf7b30aee8c93: Status 404 returned error can't find the container with id e5b24c3438cc401dd50211a6898bf330feab14f77d3a3557d83bf7b30aee8c93 Sep 18 18:14:11.949505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:11.949478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:14:11.958246 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:14:11.958216 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bae033_c422_4dda_bbaf_d4f9f52d8cdb.slice/crio-ea1094483473f8e7d25211f0b45ef1d7429dd37aead7e076c9e516b510850378 WatchSource:0}: Error finding container ea1094483473f8e7d25211f0b45ef1d7429dd37aead7e076c9e516b510850378: Status 404 returned error can't find the container with id ea1094483473f8e7d25211f0b45ef1d7429dd37aead7e076c9e516b510850378 Sep 18 18:14:12.529495 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.529452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" event={"ID":"816e406b-7b02-4bf3-b0f7-be41f02c88e5","Type":"ContainerStarted","Data":"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3"} Sep 18 18:14:12.529957 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.529528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" event={"ID":"816e406b-7b02-4bf3-b0f7-be41f02c88e5","Type":"ContainerStarted","Data":"e5b24c3438cc401dd50211a6898bf330feab14f77d3a3557d83bf7b30aee8c93"} Sep 18 18:14:12.529957 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.529561 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:12.531111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.531083 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" event={"ID":"00bae033-c422-4dda-bbaf-d4f9f52d8cdb","Type":"ContainerStarted","Data":"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13"} Sep 18 18:14:12.531280 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.531133 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:12.531280 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.531149 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" event={"ID":"00bae033-c422-4dda-bbaf-d4f9f52d8cdb","Type":"ContainerStarted","Data":"ea1094483473f8e7d25211f0b45ef1d7429dd37aead7e076c9e516b510850378"} Sep 18 18:14:12.545994 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.545949 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" podStartSLOduration=1.545935755 podStartE2EDuration="1.545935755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:12.544434275 +0000 UTC m=+1457.637089372" watchObservedRunningTime="2026-09-18 18:14:12.545935755 +0000 UTC m=+1457.638590842" Sep 18 18:14:12.561345 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:12.561304 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" podStartSLOduration=1.5612835980000002 podStartE2EDuration="1.561283598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:12.560658959 +0000 UTC m=+1457.653314076" watchObservedRunningTime="2026-09-18 18:14:12.561283598 +0000 UTC m=+1457.653938685" Sep 18 18:14:23.540342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:23.540310 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:14:23.540777 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:23.540462 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:14:43.194868 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:43.194833 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:14:45.249019 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.248982 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np"] Sep 18 18:14:45.252664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.252646 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:45.255446 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.255426 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:14:45.256315 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.256287 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:14:45.259762 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.259742 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np"] Sep 18 18:14:45.379455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.379415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np\" (UID: \"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:45.480780 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.480749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np\" (UID: \"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:45.481423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.481404 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np\" (UID: \"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:45.564499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.564423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:45.833945 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:45.833921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np"] Sep 18 18:14:45.836087 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:14:45.836060 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0419e995_d6f1_4d2e_a4b9_dd67f6198b7c.slice/crio-b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df WatchSource:0}: Error finding container b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df: Status 404 returned error can't find the container with id b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df Sep 18 18:14:46.725858 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:46.725823 2570 generic.go:356] "Generic (PLEG): container finished" podID="0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" containerID="412d361566a402f370aee83c54450e12a67b07b34cce724f30011f5cb5ea2181" exitCode=0 Sep 18 18:14:46.726257 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:46.725885 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" event={"ID":"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c","Type":"ContainerDied","Data":"412d361566a402f370aee83c54450e12a67b07b34cce724f30011f5cb5ea2181"} Sep 18 18:14:46.726257 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:46.725946 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" event={"ID":"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c","Type":"ContainerStarted","Data":"b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df"} Sep 18 18:14:47.869684 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:47.869660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:48.004162 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.004076 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca\") pod \"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c\" (UID: \"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c\") " Sep 18 18:14:48.004464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.004441 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" (UID: "0419e995-d6f1-4d2e-a4b9-dd67f6198b7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:14:48.105038 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.105004 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:48.738719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.738686 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" Sep 18 18:14:48.738719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.738712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np" event={"ID":"0419e995-d6f1-4d2e-a4b9-dd67f6198b7c","Type":"ContainerDied","Data":"b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df"} Sep 18 18:14:48.738944 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:48.738753 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0b0b7ec505d5265c2e1c39e1703beb65f3f499a7d08da191b167c8c3c8d77df" Sep 18 18:14:55.896921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:55.896877 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np"] Sep 18 18:14:55.899669 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:55.899645 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7e72d9f-4w2np"] Sep 18 18:14:56.944966 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:56.944929 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" path="/var/lib/kubelet/pods/0419e995-d6f1-4d2e-a4b9-dd67f6198b7c/volumes" Sep 18 18:14:58.214783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.214718 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" podUID="583120d8-71e3-463f-94a1-f4299fc3eff1" containerName="main" containerID="cri-o://4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007" gracePeriod=45 Sep 18 18:14:58.466997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.466945 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:14:58.600922 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.600886 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.600933 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.600987 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601042 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601121 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601214 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtfhf\" (UniqueName: \"kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf\") pod \"583120d8-71e3-463f-94a1-f4299fc3eff1\" (UID: \"583120d8-71e3-463f-94a1-f4299fc3eff1\") " Sep 18 18:14:58.601325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601216 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:14:58.601325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601279 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache" (OuterVolumeSpecName: "model-cache") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:14:58.601532 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601418 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home" (OuterVolumeSpecName: "home") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:14:58.601586 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601568 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-home\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.601628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601592 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-tmp-dir\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.601628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.601604 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-model-cache\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.603385 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.603362 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf" (OuterVolumeSpecName: "kube-api-access-gtfhf") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "kube-api-access-gtfhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:14:58.603628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.603593 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm" (OuterVolumeSpecName: "dshm") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:14:58.603628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.603606 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "583120d8-71e3-463f-94a1-f4299fc3eff1" (UID: "583120d8-71e3-463f-94a1-f4299fc3eff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 18:14:58.702242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.702174 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583120d8-71e3-463f-94a1-f4299fc3eff1-dshm\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.702242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.702241 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtfhf\" (UniqueName: \"kubernetes.io/projected/583120d8-71e3-463f-94a1-f4299fc3eff1-kube-api-access-gtfhf\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.702431 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.702257 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583120d8-71e3-463f-94a1-f4299fc3eff1-tls-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:14:58.797920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.797840 2570 generic.go:356] "Generic (PLEG): container finished" podID="583120d8-71e3-463f-94a1-f4299fc3eff1" containerID="4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007" exitCode=0 Sep 18 18:14:58.797920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.797881 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" Sep 18 18:14:58.798111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.797926 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" event={"ID":"583120d8-71e3-463f-94a1-f4299fc3eff1","Type":"ContainerDied","Data":"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007"} Sep 18 18:14:58.798111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.797966 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k" event={"ID":"583120d8-71e3-463f-94a1-f4299fc3eff1","Type":"ContainerDied","Data":"b03247a603c9bb88690eb20d3175575aee35391c43a847f69698d71785f7df2f"} Sep 18 18:14:58.798111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.797986 2570 scope.go:117] "RemoveContainer" containerID="4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007" Sep 18 18:14:58.807444 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.807431 2570 scope.go:117] "RemoveContainer" containerID="4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007" Sep 18 18:14:58.807720 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:14:58.807699 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007\": container with ID starting with 4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007 not found: ID does not exist" containerID="4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007" Sep 18 18:14:58.807768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.807728 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007"} err="failed to get container status \"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007\": rpc error: code = NotFound desc = could not find container \"4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007\": container with ID starting with 4961cc979f412e507d74a4f5e9306437141111d98a385be2c0ff30dce3fd2007 not found: ID does not exist" Sep 18 18:14:58.819928 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.819904 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:14:58.822107 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.822085 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/e2e-conflict-a-e4d74f58-kserve-57d5589864-tf74k"] Sep 18 18:14:58.941780 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:14:58.941750 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583120d8-71e3-463f-94a1-f4299fc3eff1" path="/var/lib/kubelet/pods/583120d8-71e3-463f-94a1-f4299fc3eff1/volumes" Sep 18 18:15:00.142946 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.142916 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5"] Sep 18 18:15:00.143349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143291 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583120d8-71e3-463f-94a1-f4299fc3eff1" containerName="main" Sep 18 18:15:00.143349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143302 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="583120d8-71e3-463f-94a1-f4299fc3eff1" containerName="main" Sep 18 18:15:00.143349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143316 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" containerName="revoke-keys" Sep 18 18:15:00.143349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143322 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" containerName="revoke-keys" Sep 18 18:15:00.143479 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143384 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="583120d8-71e3-463f-94a1-f4299fc3eff1" containerName="main" Sep 18 18:15:00.143479 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.143395 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0419e995-d6f1-4d2e-a4b9-dd67f6198b7c" containerName="revoke-keys" Sep 18 18:15:00.147723 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.147698 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" Sep 18 18:15:00.148438 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.148422 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt"] Sep 18 18:15:00.150111 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.150093 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:15:00.151513 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.151492 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr"] Sep 18 18:15:00.151656 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.151635 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" Sep 18 18:15:00.154644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.154622 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9"] Sep 18 18:15:00.154877 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.154682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" Sep 18 18:15:00.158359 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.158340 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5"] Sep 18 18:15:00.158562 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.158546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" Sep 18 18:15:00.162012 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.161991 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt"] Sep 18 18:15:00.162602 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.162545 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" Sep 18 18:15:00.165617 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.165414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr"] Sep 18 18:15:00.169403 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.169381 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9"] Sep 18 18:15:00.173590 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.173498 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" Sep 18 18:15:00.213052 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.213027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" Sep 18 18:15:00.221621 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.221594 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" Sep 18 18:15:00.454997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.454973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5"] Sep 18 18:15:00.456731 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:15:00.456698 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ee40ef_207b_4cd9_a304_12a9fc6eb10d.slice/crio-caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63 WatchSource:0}: Error finding container caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63: Status 404 returned error can't find the container with id caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63 Sep 18 18:15:00.465716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.465689 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt"] Sep 18 18:15:00.469497 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:15:00.469474 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf079112b_b34c_4d57_9b32_a1b4300afd1c.slice/crio-fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870 WatchSource:0}: Error finding container fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870: Status 404 returned error can't find the container with id fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870 Sep 18 18:15:00.535377 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.535337 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr"] Sep 18 18:15:00.536682 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:15:00.536590 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673f3316_4e21_4672_b3c4_6b893a5cd348.slice/crio-7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073 WatchSource:0}: Error finding container 7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073: Status 404 returned error can't find the container with id 7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073 Sep 18 18:15:00.538622 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:15:00.538597 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a948cea_08af_4d0d_a1f4_f03e062df5c0.slice/crio-862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8 WatchSource:0}: Error finding container 862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8: Status 404 returned error can't find the container with id 862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8 Sep 18 18:15:00.538714 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.538654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9"] Sep 18 18:15:00.811421 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.811393 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" event={"ID":"f079112b-b34c-4d57-9b32-a1b4300afd1c","Type":"ContainerStarted","Data":"fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870"} Sep 18 18:15:00.812995 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.812972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" event={"ID":"673f3316-4e21-4672-b3c4-6b893a5cd348","Type":"ContainerStarted","Data":"9a39062a7a3f4451da45a7efc457986ec3a378c0ba8edd9c8bcfee1f16b42a75"} Sep 18 18:15:00.813100 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.813014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" event={"ID":"673f3316-4e21-4672-b3c4-6b893a5cd348","Type":"ContainerStarted","Data":"7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073"} Sep 18 18:15:00.814321 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.814298 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" event={"ID":"49ee40ef-207b-4cd9-a304-12a9fc6eb10d","Type":"ContainerStarted","Data":"caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63"} Sep 18 18:15:00.816005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.815982 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" event={"ID":"6a948cea-08af-4d0d-a1f4-f03e062df5c0","Type":"ContainerStarted","Data":"c00363f15699e0614c6e5d3cadb61a4442fd22ac5e0b7b36fe59c59817eab27f"} Sep 18 18:15:00.816105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.816026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" event={"ID":"6a948cea-08af-4d0d-a1f4-f03e062df5c0","Type":"ContainerStarted","Data":"862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8"} Sep 18 18:15:00.828557 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.828519 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" podStartSLOduration=0.828505075 podStartE2EDuration="828.505075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:00.827262137 +0000 UTC m=+1505.919917245" watchObservedRunningTime="2026-09-18 18:15:00.828505075 +0000 UTC m=+1505.921160161" Sep 18 18:15:00.842402 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:00.842365 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" podStartSLOduration=0.842351159 podStartE2EDuration="842.351159ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:00.841989216 +0000 UTC m=+1505.934644323" watchObservedRunningTime="2026-09-18 18:15:00.842351159 +0000 UTC m=+1505.935006243" Sep 18 18:15:01.823538 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.823449 2570 generic.go:356] "Generic (PLEG): container finished" podID="49ee40ef-207b-4cd9-a304-12a9fc6eb10d" containerID="7849ce34fa1c1fded1003f878d4bc59cbecdccfff056b0e9748e837773465a3e" exitCode=0 Sep 18 18:15:01.823888 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.823539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" event={"ID":"49ee40ef-207b-4cd9-a304-12a9fc6eb10d","Type":"ContainerDied","Data":"7849ce34fa1c1fded1003f878d4bc59cbecdccfff056b0e9748e837773465a3e"} Sep 18 18:15:01.825203 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.825155 2570 generic.go:356] "Generic (PLEG): container finished" podID="6a948cea-08af-4d0d-a1f4-f03e062df5c0" containerID="c00363f15699e0614c6e5d3cadb61a4442fd22ac5e0b7b36fe59c59817eab27f" exitCode=0 Sep 18 18:15:01.825329 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.825213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" event={"ID":"6a948cea-08af-4d0d-a1f4-f03e062df5c0","Type":"ContainerDied","Data":"c00363f15699e0614c6e5d3cadb61a4442fd22ac5e0b7b36fe59c59817eab27f"} Sep 18 18:15:01.826716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.826692 2570 generic.go:356] "Generic (PLEG): container finished" podID="f079112b-b34c-4d57-9b32-a1b4300afd1c" containerID="16022ab15e5d74b080c831f74aa9f83cbfca32afe821182c8c46680961521dc8" exitCode=0 Sep 18 18:15:01.826818 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.826766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" event={"ID":"f079112b-b34c-4d57-9b32-a1b4300afd1c","Type":"ContainerDied","Data":"16022ab15e5d74b080c831f74aa9f83cbfca32afe821182c8c46680961521dc8"} Sep 18 18:15:01.828221 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.828177 2570 generic.go:356] "Generic (PLEG): container finished" podID="673f3316-4e21-4672-b3c4-6b893a5cd348" containerID="9a39062a7a3f4451da45a7efc457986ec3a378c0ba8edd9c8bcfee1f16b42a75" exitCode=0 Sep 18 18:15:01.828322 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:01.828228 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" event={"ID":"673f3316-4e21-4672-b3c4-6b893a5cd348","Type":"ContainerDied","Data":"9a39062a7a3f4451da45a7efc457986ec3a378c0ba8edd9c8bcfee1f16b42a75"} Sep 18 18:15:03.041432 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.041408 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" Sep 18 18:15:03.050852 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.050830 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" Sep 18 18:15:03.053931 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.053912 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" Sep 18 18:15:03.057500 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.057481 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" Sep 18 18:15:03.841967 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.841936 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" Sep 18 18:15:03.842145 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.841972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt" event={"ID":"f079112b-b34c-4d57-9b32-a1b4300afd1c","Type":"ContainerDied","Data":"fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870"} Sep 18 18:15:03.842145 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.842068 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc1e699d436f768ab11cef3913f2f32daeab837add055705508110b503f1e870" Sep 18 18:15:03.843388 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.843356 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" Sep 18 18:15:03.843388 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.843368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr" event={"ID":"673f3316-4e21-4672-b3c4-6b893a5cd348","Type":"ContainerDied","Data":"7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073"} Sep 18 18:15:03.843589 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.843400 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7321377598a4b6c65f75d9f6206224820ebf105a9a3336516058a00c7bb88073" Sep 18 18:15:03.844775 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.844755 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" Sep 18 18:15:03.844775 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.844769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5" event={"ID":"49ee40ef-207b-4cd9-a304-12a9fc6eb10d","Type":"ContainerDied","Data":"caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63"} Sep 18 18:15:03.844921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.844800 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caa821287e26b04377f55335c6f65fb455f2f84b3f014954cb64a45785d5ef63" Sep 18 18:15:03.846380 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.846344 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" event={"ID":"6a948cea-08af-4d0d-a1f4-f03e062df5c0","Type":"ContainerDied","Data":"862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8"} Sep 18 18:15:03.846551 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.846385 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862406c61141e09ee60ddc9578fc82e7a4c0bfc2263a32f84cd0858c896620f8" Sep 18 18:15:03.846551 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:03.846354 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9" Sep 18 18:15:57.181458 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:57.181427 2570 scope.go:117] "RemoveContainer" containerID="2f667297fc86502c20dec609e3198e710b57e6abd34162b60dd6a5ecd2caa321" Sep 18 18:15:57.190151 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:15:57.190135 2570 scope.go:117] "RemoveContainer" containerID="f9e8fa852ad753b8c9be4ae93e1fab911fc37350ccee8390d2470655a85ecad5" Sep 18 18:16:10.592716 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:10.592676 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:16:10.593248 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:10.592944 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" containerID="cri-o://d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b" gracePeriod=30 Sep 18 18:16:15.848976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:15.848952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:16:16.008567 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008533 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.008768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008577 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6mbj\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.008768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.008768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008651 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.008768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.008768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008753 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.009053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008786 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.009053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008817 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.009053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008855 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token\") pod \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\" (UID: \"4e6ac893-e82b-47cf-83f0-1834c541ea6e\") " Sep 18 18:16:16.009053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.008975 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:16:16.009289 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009142 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:16:16.009289 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009211 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:16:16.009404 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009358 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:16:16.009518 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009456 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:16:16.009518 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009506 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.009666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009533 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.009666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.009550 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.010887 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.010866 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj" (OuterVolumeSpecName: "kube-api-access-m6mbj") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "kube-api-access-m6mbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:16:16.010887 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.010879 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:16:16.011168 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.011152 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:16:16.011353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.011336 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e6ac893-e82b-47cf-83f0-1834c541ea6e" (UID: "4e6ac893-e82b-47cf-83f0-1834c541ea6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:16:16.110448 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110418 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.110448 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110446 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.110448 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110456 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.110667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110465 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.110667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110474 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mbj\" (UniqueName: \"kubernetes.io/projected/4e6ac893-e82b-47cf-83f0-1834c541ea6e-kube-api-access-m6mbj\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.110667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.110482 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ac893-e82b-47cf-83f0-1834c541ea6e-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:16:16.261600 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.261514 2570 generic.go:356] "Generic (PLEG): container finished" podID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerID="d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b" exitCode=0 Sep 18 18:16:16.261600 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.261555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerDied","Data":"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b"} Sep 18 18:16:16.261778 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.261616 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" event={"ID":"4e6ac893-e82b-47cf-83f0-1834c541ea6e","Type":"ContainerDied","Data":"5f68739ebd64e95c64951077d56d730ec7357edd1074f44a70a43aca593474af"} Sep 18 18:16:16.261778 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.261640 2570 scope.go:117] "RemoveContainer" containerID="d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b" Sep 18 18:16:16.261778 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.261571 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl" Sep 18 18:16:16.271316 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.271299 2570 scope.go:117] "RemoveContainer" containerID="1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7" Sep 18 18:16:16.279749 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.279732 2570 scope.go:117] "RemoveContainer" containerID="d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b" Sep 18 18:16:16.280012 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:16:16.279994 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b\": container with ID starting with d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b not found: ID does not exist" containerID="d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b" Sep 18 18:16:16.280062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.280024 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b"} err="failed to get container status \"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b\": rpc error: code = NotFound desc = could not find container \"d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b\": container with ID starting with d3eb2b2c0add9363c6f3c8b8b47fb44efcb0cfb8277f67e499eb7f0f6a8e017b not found: ID does not exist" Sep 18 18:16:16.280062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.280046 2570 scope.go:117] "RemoveContainer" containerID="1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7" Sep 18 18:16:16.280468 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:16:16.280433 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7\": container with ID starting with 1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7 not found: ID does not exist" containerID="1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7" Sep 18 18:16:16.280565 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.280480 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7"} err="failed to get container status \"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7\": rpc error: code = NotFound desc = could not find container \"1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7\": container with ID starting with 1af95d92905b7e3984622cd644b914ca1b40e2a55fab529bbb7f4cfd5c8280b7 not found: ID does not exist" Sep 18 18:16:16.283346 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.283325 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:16:16.287220 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.287199 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7e72d9f-openshift-default-84c6549556-sprfl"] Sep 18 18:16:16.942038 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:16:16.942006 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" path="/var/lib/kubelet/pods/4e6ac893-e82b-47cf-83f0-1834c541ea6e/volumes" Sep 18 18:17:05.019302 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019262 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019614 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673f3316-4e21-4672-b3c4-6b893a5cd348" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019635 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="673f3316-4e21-4672-b3c4-6b893a5cd348" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019653 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a948cea-08af-4d0d-a1f4-f03e062df5c0" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019658 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a948cea-08af-4d0d-a1f4-f03e062df5c0" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019668 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019674 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019683 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019688 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019694 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f079112b-b34c-4d57-9b32-a1b4300afd1c" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019699 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f079112b-b34c-4d57-9b32-a1b4300afd1c" containerName="cleanup" Sep 18 18:17:05.019703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019706 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ee40ef-207b-4cd9-a304-12a9fc6eb10d" containerName="cleanup" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019711 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ee40ef-207b-4cd9-a304-12a9fc6eb10d" containerName="cleanup" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019772 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a948cea-08af-4d0d-a1f4-f03e062df5c0" containerName="cleanup" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019780 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019787 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="673f3316-4e21-4672-b3c4-6b893a5cd348" containerName="cleanup" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019792 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f079112b-b34c-4d57-9b32-a1b4300afd1c" containerName="cleanup" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019798 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6ac893-e82b-47cf-83f0-1834c541ea6e" containerName="istio-proxy" Sep 18 18:17:05.020178 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.019803 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ee40ef-207b-4cd9-a304-12a9fc6eb10d" containerName="cleanup" Sep 18 18:17:05.023390 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.023370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.025828 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.025805 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e9a4f9-openshift-default-dockercfg-bxgc8\"" Sep 18 18:17:05.037891 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.037861 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:17:05.157523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtlz\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.157913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.157872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259132 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259132 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtlz\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259423 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259397 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259651 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259651 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259775 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259797 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.259850 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.260009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.259989 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.260090 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.260073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.261592 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.261564 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.261834 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.261814 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.267208 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.267165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.267443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.267424 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtlz\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz\") pod \"e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.334128 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.334021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:05.617729 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.617696 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:17:05.619414 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:17:05.619386 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2afadaf_500a_4db9_a647_72995df317d8.slice/crio-8dfcb9606af4a91a5c67edcbd54d371ddc238ef39ff0ea578ff0c15b35569fa9 WatchSource:0}: Error finding container 8dfcb9606af4a91a5c67edcbd54d371ddc238ef39ff0ea578ff0c15b35569fa9: Status 404 returned error can't find the container with id 8dfcb9606af4a91a5c67edcbd54d371ddc238ef39ff0ea578ff0c15b35569fa9 Sep 18 18:17:05.621728 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.621699 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:05.621807 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.621765 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:05.621807 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:05.621795 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:06.549703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:06.549664 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerStarted","Data":"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca"} Sep 18 18:17:06.550083 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:06.549723 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerStarted","Data":"8dfcb9606af4a91a5c67edcbd54d371ddc238ef39ff0ea578ff0c15b35569fa9"} Sep 18 18:17:06.570488 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:06.570435 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podStartSLOduration=2.570419085 podStartE2EDuration="2.570419085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:06.569355685 +0000 UTC m=+1631.662010774" watchObservedRunningTime="2026-09-18 18:17:06.570419085 +0000 UTC m=+1631.663074173" Sep 18 18:17:07.334548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:07.334520 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:07.339544 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:07.339521 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:07.555972 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:07.555943 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:07.556819 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:07.556799 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:12.790660 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.790625 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:17:12.794148 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.794128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:12.796534 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.796514 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e9a4f9-dockercfg-nw4s6\"" Sep 18 18:17:12.796625 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.796516 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e9a4f9\"" Sep 18 18:17:12.803427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.803400 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:17:12.934594 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.934554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:12.934772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:12.934628 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn28\" (UniqueName: \"kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.035154 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.035121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.035337 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.035223 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn28\" (UniqueName: \"kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.035822 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.035803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.043092 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.043031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn28\" (UniqueName: \"kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28\") pod \"payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.106118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.106088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.378348 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.378323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:17:13.380230 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:17:13.380172 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a2726_c798_4f05_9a84_4e781d48859b.slice/crio-146de360c4d5f5d17dcc6fc571110004dec4a6b912d8cd5dbc25d0aff35e6b92 WatchSource:0}: Error finding container 146de360c4d5f5d17dcc6fc571110004dec4a6b912d8cd5dbc25d0aff35e6b92: Status 404 returned error can't find the container with id 146de360c4d5f5d17dcc6fc571110004dec4a6b912d8cd5dbc25d0aff35e6b92 Sep 18 18:17:13.596386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.596349 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" event={"ID":"8a7a2726-c798-4f05-9a84-4e781d48859b","Type":"ContainerStarted","Data":"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d"} Sep 18 18:17:13.596573 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.596412 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" event={"ID":"8a7a2726-c798-4f05-9a84-4e781d48859b","Type":"ContainerStarted","Data":"146de360c4d5f5d17dcc6fc571110004dec4a6b912d8cd5dbc25d0aff35e6b92"} Sep 18 18:17:13.596573 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.596474 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:13.617865 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:13.617811 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" podStartSLOduration=1.617797361 podStartE2EDuration="1.617797361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:13.614743916 +0000 UTC m=+1638.707399024" watchObservedRunningTime="2026-09-18 18:17:13.617797361 +0000 UTC m=+1638.710452448" Sep 18 18:17:24.604932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:24.604898 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:17:37.557097 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.557002 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:37.557097 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.557058 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:37.745577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.745543 2570 generic.go:356] "Generic (PLEG): container finished" podID="d2afadaf-500a-4db9-a647-72995df317d8" containerID="12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca" exitCode=0 Sep 18 18:17:37.745746 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.745615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerDied","Data":"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca"} Sep 18 18:17:37.746039 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.746025 2570 scope.go:117] "RemoveContainer" containerID="12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca" Sep 18 18:17:37.747264 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.747240 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:37.747335 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.747313 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:37.747374 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:37.747345 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:17:38.753739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:38.753707 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerStarted","Data":"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377"} Sep 18 18:17:39.335106 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:39.335068 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:39.335350 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:39.335328 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:39.335406 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:39.335378 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:40.335251 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:40.335213 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:40.335712 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:40.335278 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:41.335083 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:41.335051 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:41.335310 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:41.335106 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:42.334900 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:42.334864 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:42.335082 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:42.334921 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:43.334691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:43.334650 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:43.335163 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:43.334720 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:44.335135 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:44.335099 2570 patch_prober.go:28] interesting pod/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 18 18:17:44.335528 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:44.335163 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 18 18:17:45.341377 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:45.341346 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:45.341805 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:45.341753 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:45.342533 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:45.342514 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:17:47.382932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.382904 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:17:47.386191 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.386166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.388789 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.388768 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-005032\"" Sep 18 18:17:47.398752 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.398726 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:17:47.428670 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.428644 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:17:47.431280 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.431252 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.434381 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.434363 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-005032-dockercfg-9lzlc\"" Sep 18 18:17:47.441281 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.441259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lcl\" (UniqueName: \"kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.441377 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.441313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.447820 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.447798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:17:47.541918 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.541874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.542150 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.541992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.542150 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.542046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s85z\" (UniqueName: \"kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.542150 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.542094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lcl\" (UniqueName: \"kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.543616 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.543588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.553691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.553624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lcl\" (UniqueName: \"kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl\") pod \"payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.643398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.643315 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s85z\" (UniqueName: \"kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.643552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.643418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.644068 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.644049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.651469 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.651439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s85z\" (UniqueName: \"kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z\") pod \"payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.698646 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.698619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:47.740603 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.740575 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:47.970938 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:47.970909 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:17:47.972277 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:17:47.972211 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5576f95_ea83_47de_865d_4a04f8b77f01.slice/crio-a92f9754b64f139bc2253b904667d362378f5d6a73bdc9edf89e66a084aaeb95 WatchSource:0}: Error finding container a92f9754b64f139bc2253b904667d362378f5d6a73bdc9edf89e66a084aaeb95: Status 404 returned error can't find the container with id a92f9754b64f139bc2253b904667d362378f5d6a73bdc9edf89e66a084aaeb95 Sep 18 18:17:48.030007 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.029980 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:17:48.031651 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:17:48.031628 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae258829_9efe_4fd5_b3bf_b81e8cb57561.slice/crio-96a496af751baea0c416666405dbf02d6186ad42c1b95d8025a6b4607539ee54 WatchSource:0}: Error finding container 96a496af751baea0c416666405dbf02d6186ad42c1b95d8025a6b4607539ee54: Status 404 returned error can't find the container with id 96a496af751baea0c416666405dbf02d6186ad42c1b95d8025a6b4607539ee54 Sep 18 18:17:48.827261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.827223 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" event={"ID":"a5576f95-ea83-47de-865d-4a04f8b77f01","Type":"ContainerStarted","Data":"1e5d434ef6626695c38fedc85bdc6c6c0f53dee9aec3b98d5d9db33710cfc08a"} Sep 18 18:17:48.827725 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.827290 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" event={"ID":"a5576f95-ea83-47de-865d-4a04f8b77f01","Type":"ContainerStarted","Data":"a92f9754b64f139bc2253b904667d362378f5d6a73bdc9edf89e66a084aaeb95"} Sep 18 18:17:48.827725 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.827351 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:48.829036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.829014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" event={"ID":"ae258829-9efe-4fd5-b3bf-b81e8cb57561","Type":"ContainerStarted","Data":"546f25216dafd84c728edd3b5dda41fcd13e17c54bb37f6c412154b3e04b845a"} Sep 18 18:17:48.829161 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.829047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" event={"ID":"ae258829-9efe-4fd5-b3bf-b81e8cb57561","Type":"ContainerStarted","Data":"96a496af751baea0c416666405dbf02d6186ad42c1b95d8025a6b4607539ee54"} Sep 18 18:17:48.829161 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.829081 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:17:48.846528 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.846488 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" podStartSLOduration=1.846474643 podStartE2EDuration="1.846474643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:48.844301548 +0000 UTC m=+1673.936956656" watchObservedRunningTime="2026-09-18 18:17:48.846474643 +0000 UTC m=+1673.939129728" Sep 18 18:17:48.861011 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:48.860963 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" podStartSLOduration=1.8609508350000001 podStartE2EDuration="1.860950835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:48.860333167 +0000 UTC m=+1673.952988278" watchObservedRunningTime="2026-09-18 18:17:48.860950835 +0000 UTC m=+1673.953605970" Sep 18 18:17:57.236427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:57.236398 2570 scope.go:117] "RemoveContainer" containerID="0dc12d08542b64b330d4cea0166cef7195bcf25ef609dad87f95cf9b1c043ac5" Sep 18 18:17:59.836852 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:59.836823 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:17:59.837328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:17:59.837101 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:18:15.012958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.012881 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb"] Sep 18 18:18:15.017928 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.017902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:15.043901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.043875 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:18:15.044060 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.044044 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:18:15.072919 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.072887 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb"] Sep 18 18:18:15.092125 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.092087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb\" (UID: \"2afa7cbb-882f-41b3-9c51-a96923402508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:15.193550 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.193515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb\" (UID: \"2afa7cbb-882f-41b3-9c51-a96923402508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:15.194099 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.194081 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb\" (UID: \"2afa7cbb-882f-41b3-9c51-a96923402508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:15.327581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.327492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:15.612594 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.612560 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb"] Sep 18 18:18:15.614204 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:18:15.614162 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afa7cbb_882f_41b3_9c51_a96923402508.slice/crio-8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515 WatchSource:0}: Error finding container 8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515: Status 404 returned error can't find the container with id 8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515 Sep 18 18:18:15.982905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.982875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" event={"ID":"2afa7cbb-882f-41b3-9c51-a96923402508","Type":"ContainerStarted","Data":"d7a76780d0cfb2ae101adab71f47c13883bb423b43d6513b3ff15dc2b2ac6a26"} Sep 18 18:18:15.983068 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.982925 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" event={"ID":"2afa7cbb-882f-41b3-9c51-a96923402508","Type":"ContainerStarted","Data":"8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515"} Sep 18 18:18:15.998914 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:15.998863 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" podStartSLOduration=1.998846862 podStartE2EDuration="1.998846862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:18:15.998422634 +0000 UTC m=+1701.091077748" watchObservedRunningTime="2026-09-18 18:18:15.998846862 +0000 UTC m=+1701.091501948" Sep 18 18:18:16.992938 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:16.992897 2570 generic.go:356] "Generic (PLEG): container finished" podID="2afa7cbb-882f-41b3-9c51-a96923402508" containerID="d7a76780d0cfb2ae101adab71f47c13883bb423b43d6513b3ff15dc2b2ac6a26" exitCode=0 Sep 18 18:18:16.993383 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:16.992993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" event={"ID":"2afa7cbb-882f-41b3-9c51-a96923402508","Type":"ContainerDied","Data":"d7a76780d0cfb2ae101adab71f47c13883bb423b43d6513b3ff15dc2b2ac6a26"} Sep 18 18:18:18.131937 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:18.131910 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:18.220626 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:18.220595 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca\") pod \"2afa7cbb-882f-41b3-9c51-a96923402508\" (UID: \"2afa7cbb-882f-41b3-9c51-a96923402508\") " Sep 18 18:18:18.220969 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:18.220945 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2afa7cbb-882f-41b3-9c51-a96923402508" (UID: "2afa7cbb-882f-41b3-9c51-a96923402508"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:18.321574 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:18.321495 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa7cbb-882f-41b3-9c51-a96923402508-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:19.005948 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:19.005923 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" Sep 18 18:18:19.006113 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:19.005959 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb" event={"ID":"2afa7cbb-882f-41b3-9c51-a96923402508","Type":"ContainerDied","Data":"8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515"} Sep 18 18:18:19.006113 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:19.006005 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf2e71fe542cbaa6079c426941c088cccf924f266e170d15c3f847271013515" Sep 18 18:18:30.525615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.525581 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:18:30.526500 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.526455 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" podUID="8a7a2726-c798-4f05-9a84-4e781d48859b" containerName="payload-pre-processing" containerID="cri-o://ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d" gracePeriod=30 Sep 18 18:18:30.767432 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.767396 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb"] Sep 18 18:18:30.774418 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.774375 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e9a4f9-m45sb"] Sep 18 18:18:30.801541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.801515 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:18:30.934276 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.934240 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume\") pod \"8a7a2726-c798-4f05-9a84-4e781d48859b\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " Sep 18 18:18:30.934476 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.934338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vn28\" (UniqueName: \"kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28\") pod \"8a7a2726-c798-4f05-9a84-4e781d48859b\" (UID: \"8a7a2726-c798-4f05-9a84-4e781d48859b\") " Sep 18 18:18:30.934624 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.934596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a7a2726-c798-4f05-9a84-4e781d48859b" (UID: "8a7a2726-c798-4f05-9a84-4e781d48859b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:30.934704 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.934686 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a2726-c798-4f05-9a84-4e781d48859b-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:30.936413 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.936385 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28" (OuterVolumeSpecName: "kube-api-access-9vn28") pod "8a7a2726-c798-4f05-9a84-4e781d48859b" (UID: "8a7a2726-c798-4f05-9a84-4e781d48859b"). InnerVolumeSpecName "kube-api-access-9vn28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:18:30.942638 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:30.942612 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2afa7cbb-882f-41b3-9c51-a96923402508" path="/var/lib/kubelet/pods/2afa7cbb-882f-41b3-9c51-a96923402508/volumes" Sep 18 18:18:31.035691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.035623 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vn28\" (UniqueName: \"kubernetes.io/projected/8a7a2726-c798-4f05-9a84-4e781d48859b-kube-api-access-9vn28\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:31.078140 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.078106 2570 generic.go:356] "Generic (PLEG): container finished" podID="8a7a2726-c798-4f05-9a84-4e781d48859b" containerID="ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d" exitCode=0 Sep 18 18:18:31.078297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.078154 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" Sep 18 18:18:31.078297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.078154 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" event={"ID":"8a7a2726-c798-4f05-9a84-4e781d48859b","Type":"ContainerDied","Data":"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d"} Sep 18 18:18:31.078297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.078246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx" event={"ID":"8a7a2726-c798-4f05-9a84-4e781d48859b","Type":"ContainerDied","Data":"146de360c4d5f5d17dcc6fc571110004dec4a6b912d8cd5dbc25d0aff35e6b92"} Sep 18 18:18:31.078297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.078279 2570 scope.go:117] "RemoveContainer" containerID="ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d" Sep 18 18:18:31.088149 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.088130 2570 scope.go:117] "RemoveContainer" containerID="ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d" Sep 18 18:18:31.088465 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:18:31.088446 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d\": container with ID starting with ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d not found: ID does not exist" containerID="ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d" Sep 18 18:18:31.088531 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.088477 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d"} err="failed to get container status \"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d\": rpc error: code = NotFound desc = could not find container \"ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d\": container with ID starting with ece10a91a45cd5a8e12b46a2c448cdd5835eed5b0e596912dfb591ebb9f37a4d not found: ID does not exist" Sep 18 18:18:31.103824 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.103802 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:18:31.106115 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:31.106096 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e9a4f9-77dc75b8f8-9mdcx"] Sep 18 18:18:32.943530 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:32.943497 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7a2726-c798-4f05-9a84-4e781d48859b" path="/var/lib/kubelet/pods/8a7a2726-c798-4f05-9a84-4e781d48859b/volumes" Sep 18 18:18:33.488369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488329 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c"] Sep 18 18:18:33.488795 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488778 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7a2726-c798-4f05-9a84-4e781d48859b" containerName="payload-pre-processing" Sep 18 18:18:33.488883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488798 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7a2726-c798-4f05-9a84-4e781d48859b" containerName="payload-pre-processing" Sep 18 18:18:33.488883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488824 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afa7cbb-882f-41b3-9c51-a96923402508" containerName="revoke-keys" Sep 18 18:18:33.488883 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488833 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afa7cbb-882f-41b3-9c51-a96923402508" containerName="revoke-keys" Sep 18 18:18:33.489039 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488924 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7a2726-c798-4f05-9a84-4e781d48859b" containerName="payload-pre-processing" Sep 18 18:18:33.489039 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.488937 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afa7cbb-882f-41b3-9c51-a96923402508" containerName="revoke-keys" Sep 18 18:18:33.493268 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.493246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:33.495566 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.495547 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:18:33.495675 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.495547 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:18:33.501017 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.500986 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c"] Sep 18 18:18:33.560029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.559999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c\" (UID: \"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:33.661412 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.661377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c\" (UID: \"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:33.662051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.662025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c\" (UID: \"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:33.805959 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:33.805864 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:34.090299 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:34.090268 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c"] Sep 18 18:18:34.090880 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:18:34.090857 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf99df7_a5bc_4dc5_8c56_e17e6d2eb3a9.slice/crio-7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674 WatchSource:0}: Error finding container 7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674: Status 404 returned error can't find the container with id 7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674 Sep 18 18:18:34.098685 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:34.098661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" event={"ID":"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9","Type":"ContainerStarted","Data":"7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674"} Sep 18 18:18:35.106548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:35.106451 2570 generic.go:356] "Generic (PLEG): container finished" podID="3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" containerID="c8874f2b149fc98abe95778dbaf168560d6226743202eb87aae84d9f083e67a6" exitCode=0 Sep 18 18:18:35.106548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:35.106485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" event={"ID":"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9","Type":"ContainerDied","Data":"c8874f2b149fc98abe95778dbaf168560d6226743202eb87aae84d9f083e67a6"} Sep 18 18:18:36.250927 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:36.250903 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:36.387051 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:36.386963 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca\") pod \"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9\" (UID: \"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9\") " Sep 18 18:18:36.387395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:36.387372 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" (UID: "3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:36.488577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:36.488542 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:37.119139 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.119110 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" Sep 18 18:18:37.119324 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.119157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c" event={"ID":"3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9","Type":"ContainerDied","Data":"7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674"} Sep 18 18:18:37.119324 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.119230 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e664fdd1a9c24db7afc4b9ab7becab73764a4a015a4a171c8af1de55f45f674" Sep 18 18:18:37.194876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.194841 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:18:37.195054 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.195038 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" containerID="cri-o://b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377" gracePeriod=30 Sep 18 18:18:37.571249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.571214 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx"] Sep 18 18:18:37.571632 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.571595 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" containerName="revoke-keys" Sep 18 18:18:37.571632 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.571606 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" containerName="revoke-keys" Sep 18 18:18:37.571703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.571677 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" containerName="revoke-keys" Sep 18 18:18:37.574540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.574521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:37.577753 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.577708 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:18:37.577947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.577932 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:18:37.587773 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.587750 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx"] Sep 18 18:18:37.699719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.699683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-005032-86kdx\" (UID: \"5aebdd58-0551-4fed-9bc4-70f8c51ce13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:37.800310 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.800277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-005032-86kdx\" (UID: \"5aebdd58-0551-4fed-9bc4-70f8c51ce13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:37.800942 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.800921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-005032-86kdx\" (UID: \"5aebdd58-0551-4fed-9bc4-70f8c51ce13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:37.885134 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:37.885043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:38.161844 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:38.161820 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx"] Sep 18 18:18:38.162773 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:18:38.162751 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aebdd58_0551_4fed_9bc4_70f8c51ce13f.slice/crio-fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2 WatchSource:0}: Error finding container fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2: Status 404 returned error can't find the container with id fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2 Sep 18 18:18:39.133295 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:39.133178 2570 generic.go:356] "Generic (PLEG): container finished" podID="5aebdd58-0551-4fed-9bc4-70f8c51ce13f" containerID="a04ec5738d1d71c8f17bc3438a78a7c5b8d6cdb47e37c1a71551a6387a30bd57" exitCode=0 Sep 18 18:18:39.133295 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:39.133250 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" event={"ID":"5aebdd58-0551-4fed-9bc4-70f8c51ce13f","Type":"ContainerDied","Data":"a04ec5738d1d71c8f17bc3438a78a7c5b8d6cdb47e37c1a71551a6387a30bd57"} Sep 18 18:18:39.133295 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:39.133292 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" event={"ID":"5aebdd58-0551-4fed-9bc4-70f8c51ce13f","Type":"ContainerStarted","Data":"fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2"} Sep 18 18:18:40.272680 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:40.272655 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:40.425881 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:40.425848 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca\") pod \"5aebdd58-0551-4fed-9bc4-70f8c51ce13f\" (UID: \"5aebdd58-0551-4fed-9bc4-70f8c51ce13f\") " Sep 18 18:18:40.426208 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:40.426161 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5aebdd58-0551-4fed-9bc4-70f8c51ce13f" (UID: "5aebdd58-0551-4fed-9bc4-70f8c51ce13f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:40.527203 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:40.527158 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aebdd58-0551-4fed-9bc4-70f8c51ce13f-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:41.145937 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:41.145902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" event={"ID":"5aebdd58-0551-4fed-9bc4-70f8c51ce13f","Type":"ContainerDied","Data":"fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2"} Sep 18 18:18:41.146094 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:41.145954 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe82eefcb512b01308f482ae583feb632fe859fcc0375b305c0779b54e887ba2" Sep 18 18:18:41.146094 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:41.145919 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx" Sep 18 18:18:42.458538 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.458484 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:18:42.544435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544405 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544484 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544509 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544531 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544546 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544609 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qtlz\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544653 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544695 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data\") pod \"d2afadaf-500a-4db9-a647-72995df317d8\" (UID: \"d2afadaf-500a-4db9-a647-72995df317d8\") " Sep 18 18:18:42.544926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544884 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:18:42.544926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.544920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:42.545138 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.545078 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2afadaf-500a-4db9-a647-72995df317d8-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.545138 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.545097 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.545138 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.545093 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:18:42.545311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.545156 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:18:42.545311 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.545228 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:18:42.546729 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.546710 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:18:42.546921 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.546899 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz" (OuterVolumeSpecName: "kube-api-access-7qtlz") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "kube-api-access-7qtlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:18:42.547106 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.547086 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:18:42.547153 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.547091 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2afadaf-500a-4db9-a647-72995df317d8" (UID: "d2afadaf-500a-4db9-a647-72995df317d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:18:42.646197 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646166 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2afadaf-500a-4db9-a647-72995df317d8-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646209 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646219 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646227 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646237 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtlz\" (UniqueName: \"kubernetes.io/projected/d2afadaf-500a-4db9-a647-72995df317d8-kube-api-access-7qtlz\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646248 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:42.646367 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:42.646255 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2afadaf-500a-4db9-a647-72995df317d8-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:43.159615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.159575 2570 generic.go:356] "Generic (PLEG): container finished" podID="d2afadaf-500a-4db9-a647-72995df317d8" containerID="b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377" exitCode=0 Sep 18 18:18:43.159801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.159642 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" Sep 18 18:18:43.159801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.159647 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerDied","Data":"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377"} Sep 18 18:18:43.159801 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.159779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk" event={"ID":"d2afadaf-500a-4db9-a647-72995df317d8","Type":"ContainerDied","Data":"8dfcb9606af4a91a5c67edcbd54d371ddc238ef39ff0ea578ff0c15b35569fa9"} Sep 18 18:18:43.159924 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.159815 2570 scope.go:117] "RemoveContainer" containerID="b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377" Sep 18 18:18:43.169499 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.169482 2570 scope.go:117] "RemoveContainer" containerID="12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca" Sep 18 18:18:43.178226 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.178206 2570 scope.go:117] "RemoveContainer" containerID="b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377" Sep 18 18:18:43.178497 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:18:43.178470 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377\": container with ID starting with b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377 not found: ID does not exist" containerID="b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377" Sep 18 18:18:43.178602 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.178503 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377"} err="failed to get container status \"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377\": rpc error: code = NotFound desc = could not find container \"b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377\": container with ID starting with b70f81fb2720ddbe2874a03ed785b3ca2578d4a6a6fe7140c2dff0b6e1204377 not found: ID does not exist" Sep 18 18:18:43.178602 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.178520 2570 scope.go:117] "RemoveContainer" containerID="12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca" Sep 18 18:18:43.178774 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:18:43.178755 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca\": container with ID starting with 12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca not found: ID does not exist" containerID="12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca" Sep 18 18:18:43.178813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.178783 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca"} err="failed to get container status \"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca\": rpc error: code = NotFound desc = could not find container \"12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca\": container with ID starting with 12d9f04fea5daa8030098a236982b90b452eb16f2482d5cebe7e715332f53dca not found: ID does not exist" Sep 18 18:18:43.179447 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.179427 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:18:43.186124 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:43.186098 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e9a4f9-openshift-default-99f98b648-2jszk"] Sep 18 18:18:44.942521 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:44.942480 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2afadaf-500a-4db9-a647-72995df317d8" path="/var/lib/kubelet/pods/d2afadaf-500a-4db9-a647-72995df317d8/volumes" Sep 18 18:18:49.574091 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:49.574058 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5"] Sep 18 18:18:49.576519 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:49.576497 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03e512-29829255-pnhw5"] Sep 18 18:18:49.771699 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:49.771641 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c"] Sep 18 18:18:49.774009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:49.773986 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03e512-prh2c"] Sep 18 18:18:50.943793 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.943762 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9" path="/var/lib/kubelet/pods/3cf99df7-a5bc-4dc5-8c56-e17e6d2eb3a9/volumes" Sep 18 18:18:50.944161 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.944043 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ee40ef-207b-4cd9-a304-12a9fc6eb10d" path="/var/lib/kubelet/pods/49ee40ef-207b-4cd9-a304-12a9fc6eb10d/volumes" Sep 18 18:18:50.983763 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.983725 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:18:50.983999 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.983960 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" podUID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" containerName="payload-processing" containerID="cri-o://546f25216dafd84c728edd3b5dda41fcd13e17c54bb37f6c412154b3e04b845a" gracePeriod=30 Sep 18 18:18:50.991903 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.991878 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:18:50.992135 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:50.992098 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" podUID="a5576f95-ea83-47de-865d-4a04f8b77f01" containerName="payload-pre-processing" containerID="cri-o://1e5d434ef6626695c38fedc85bdc6c6c0f53dee9aec3b98d5d9db33710cfc08a" gracePeriod=30 Sep 18 18:18:51.219443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.219145 2570 generic.go:356] "Generic (PLEG): container finished" podID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" containerID="546f25216dafd84c728edd3b5dda41fcd13e17c54bb37f6c412154b3e04b845a" exitCode=0 Sep 18 18:18:51.219443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.219319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" event={"ID":"ae258829-9efe-4fd5-b3bf-b81e8cb57561","Type":"ContainerDied","Data":"546f25216dafd84c728edd3b5dda41fcd13e17c54bb37f6c412154b3e04b845a"} Sep 18 18:18:51.225356 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.225073 2570 generic.go:356] "Generic (PLEG): container finished" podID="a5576f95-ea83-47de-865d-4a04f8b77f01" containerID="1e5d434ef6626695c38fedc85bdc6c6c0f53dee9aec3b98d5d9db33710cfc08a" exitCode=0 Sep 18 18:18:51.225356 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.225263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" event={"ID":"a5576f95-ea83-47de-865d-4a04f8b77f01","Type":"ContainerDied","Data":"1e5d434ef6626695c38fedc85bdc6c6c0f53dee9aec3b98d5d9db33710cfc08a"} Sep 18 18:18:51.276627 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.276603 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:18:51.282120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.282095 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx"] Sep 18 18:18:51.285705 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.285682 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-005032-86kdx"] Sep 18 18:18:51.290932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.290916 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:18:51.425771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.425738 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume\") pod \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " Sep 18 18:18:51.425947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.425793 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2lcl\" (UniqueName: \"kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl\") pod \"a5576f95-ea83-47de-865d-4a04f8b77f01\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " Sep 18 18:18:51.425947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.425876 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume\") pod \"a5576f95-ea83-47de-865d-4a04f8b77f01\" (UID: \"a5576f95-ea83-47de-865d-4a04f8b77f01\") " Sep 18 18:18:51.425947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.425918 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s85z\" (UniqueName: \"kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z\") pod \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\" (UID: \"ae258829-9efe-4fd5-b3bf-b81e8cb57561\") " Sep 18 18:18:51.426250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.426220 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae258829-9efe-4fd5-b3bf-b81e8cb57561" (UID: "ae258829-9efe-4fd5-b3bf-b81e8cb57561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:51.426380 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.426220 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5576f95-ea83-47de-865d-4a04f8b77f01" (UID: "a5576f95-ea83-47de-865d-4a04f8b77f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:18:51.427889 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.427865 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl" (OuterVolumeSpecName: "kube-api-access-n2lcl") pod "a5576f95-ea83-47de-865d-4a04f8b77f01" (UID: "a5576f95-ea83-47de-865d-4a04f8b77f01"). InnerVolumeSpecName "kube-api-access-n2lcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:18:51.427981 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.427961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z" (OuterVolumeSpecName: "kube-api-access-2s85z") pod "ae258829-9efe-4fd5-b3bf-b81e8cb57561" (UID: "ae258829-9efe-4fd5-b3bf-b81e8cb57561"). InnerVolumeSpecName "kube-api-access-2s85z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:18:51.527584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.527497 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae258829-9efe-4fd5-b3bf-b81e8cb57561-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:51.527584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.527526 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2lcl\" (UniqueName: \"kubernetes.io/projected/a5576f95-ea83-47de-865d-4a04f8b77f01-kube-api-access-n2lcl\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:51.527584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.527536 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5576f95-ea83-47de-865d-4a04f8b77f01-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:51.527584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:51.527544 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s85z\" (UniqueName: \"kubernetes.io/projected/ae258829-9efe-4fd5-b3bf-b81e8cb57561-kube-api-access-2s85z\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:18:52.232467 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.232439 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" Sep 18 18:18:52.232931 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.232480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn" event={"ID":"a5576f95-ea83-47de-865d-4a04f8b77f01","Type":"ContainerDied","Data":"a92f9754b64f139bc2253b904667d362378f5d6a73bdc9edf89e66a084aaeb95"} Sep 18 18:18:52.232931 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.232545 2570 scope.go:117] "RemoveContainer" containerID="1e5d434ef6626695c38fedc85bdc6c6c0f53dee9aec3b98d5d9db33710cfc08a" Sep 18 18:18:52.233905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.233819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" event={"ID":"ae258829-9efe-4fd5-b3bf-b81e8cb57561","Type":"ContainerDied","Data":"96a496af751baea0c416666405dbf02d6186ad42c1b95d8025a6b4607539ee54"} Sep 18 18:18:52.233905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.233868 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7" Sep 18 18:18:52.243503 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.243485 2570 scope.go:117] "RemoveContainer" containerID="546f25216dafd84c728edd3b5dda41fcd13e17c54bb37f6c412154b3e04b845a" Sep 18 18:18:52.257500 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.257479 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:18:52.262233 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.262213 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-005032-5b95d8f69d-5n5dn"] Sep 18 18:18:52.273756 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.273735 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:18:52.279165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.279144 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-005032-99d4d5d4f-lgzb7"] Sep 18 18:18:52.942071 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.942036 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aebdd58-0551-4fed-9bc4-70f8c51ce13f" path="/var/lib/kubelet/pods/5aebdd58-0551-4fed-9bc4-70f8c51ce13f/volumes" Sep 18 18:18:52.942997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.942951 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5576f95-ea83-47de-865d-4a04f8b77f01" path="/var/lib/kubelet/pods/a5576f95-ea83-47de-865d-4a04f8b77f01/volumes" Sep 18 18:18:52.943550 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:52.943526 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" path="/var/lib/kubelet/pods/ae258829-9efe-4fd5-b3bf-b81e8cb57561/volumes" Sep 18 18:18:56.692784 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:56.692753 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:18:56.693278 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:56.692934 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" containerID="cri-o://69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f" gracePeriod=30 Sep 18 18:18:57.258369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:57.258337 2570 scope.go:117] "RemoveContainer" containerID="f5de5c4766a2fa549311ee34b068078f65a88df0a69249c7d8fafab7d5b1d076" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.577389 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq"] Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578167 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578212 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578232 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5576f95-ea83-47de-865d-4a04f8b77f01" containerName="payload-pre-processing" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578243 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5576f95-ea83-47de-865d-4a04f8b77f01" containerName="payload-pre-processing" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578263 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578271 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578317 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" containerName="payload-processing" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578326 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" containerName="payload-processing" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578343 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aebdd58-0551-4fed-9bc4-70f8c51ce13f" containerName="revoke-keys" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578351 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aebdd58-0551-4fed-9bc4-70f8c51ce13f" containerName="revoke-keys" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578515 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae258829-9efe-4fd5-b3bf-b81e8cb57561" containerName="payload-processing" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578526 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578542 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aebdd58-0551-4fed-9bc4-70f8c51ce13f" containerName="revoke-keys" Sep 18 18:18:58.580267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.578552 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5576f95-ea83-47de-865d-4a04f8b77f01" containerName="payload-pre-processing" Sep 18 18:18:58.584046 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.584021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:18:58.587083 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.587063 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:18:58.587583 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.587566 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:18:58.595464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.595444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq\" (UID: \"7fa89c94-655c-410e-b46a-b01fa9019561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:18:58.599443 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.599419 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq"] Sep 18 18:18:58.695853 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.695821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq\" (UID: \"7fa89c94-655c-410e-b46a-b01fa9019561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:18:58.696461 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.696441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq\" (UID: \"7fa89c94-655c-410e-b46a-b01fa9019561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:18:58.894242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:58.894135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:18:59.171351 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:59.171316 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq"] Sep 18 18:18:59.172303 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:18:59.172278 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa89c94_655c_410e_b46a_b01fa9019561.slice/crio-a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c WatchSource:0}: Error finding container a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c: Status 404 returned error can't find the container with id a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c Sep 18 18:18:59.280177 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:18:59.280145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" event={"ID":"7fa89c94-655c-410e-b46a-b01fa9019561","Type":"ContainerStarted","Data":"a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c"} Sep 18 18:19:00.287314 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:00.287274 2570 generic.go:356] "Generic (PLEG): container finished" podID="7fa89c94-655c-410e-b46a-b01fa9019561" containerID="2d4d95039847e05c65db504081ec06a4001e2e54af0c4dc07c1ef1b8f69fa62b" exitCode=0 Sep 18 18:19:00.287705 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:00.287360 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" event={"ID":"7fa89c94-655c-410e-b46a-b01fa9019561","Type":"ContainerDied","Data":"2d4d95039847e05c65db504081ec06a4001e2e54af0c4dc07c1ef1b8f69fa62b"} Sep 18 18:19:01.423926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:01.423905 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:19:01.520550 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:01.520516 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca\") pod \"7fa89c94-655c-410e-b46a-b01fa9019561\" (UID: \"7fa89c94-655c-410e-b46a-b01fa9019561\") " Sep 18 18:19:01.520898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:01.520873 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fa89c94-655c-410e-b46a-b01fa9019561" (UID: "7fa89c94-655c-410e-b46a-b01fa9019561"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:01.621266 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:01.621140 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa89c94-655c-410e-b46a-b01fa9019561-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:01.936967 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:01.936943 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:19:02.125372 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125372 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125375 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125425 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125464 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125495 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125530 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125615 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125604 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket\") pod \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\" (UID: \"bf44764e-5a06-4b8f-b966-d8b3b69389f8\") " Sep 18 18:19:02.125970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125640 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:02.125970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125783 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:02.125970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125815 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:02.125970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125914 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:02.126165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125975 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:02.126165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.125991 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.126165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.126010 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.126165 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.126042 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.127812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.127782 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:02.127896 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.127862 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:19:02.128058 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.128035 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms" (OuterVolumeSpecName: "kube-api-access-hllms") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "kube-api-access-hllms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:02.128058 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.128049 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf44764e-5a06-4b8f-b966-d8b3b69389f8" (UID: "bf44764e-5a06-4b8f-b966-d8b3b69389f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226511 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hllms\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-kube-api-access-hllms\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226540 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226549 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226557 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226564 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf44764e-5a06-4b8f-b966-d8b3b69389f8-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.226593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.226574 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf44764e-5a06-4b8f-b966-d8b3b69389f8-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:02.300362 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.300331 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" Sep 18 18:19:02.300520 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.300367 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq" event={"ID":"7fa89c94-655c-410e-b46a-b01fa9019561","Type":"ContainerDied","Data":"a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c"} Sep 18 18:19:02.300520 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.300418 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4175029cfab7e3c2d904839f6eb43c9dcf8d226924bc7114b33fafe31cfb94c" Sep 18 18:19:02.302168 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.302145 2570 generic.go:356] "Generic (PLEG): container finished" podID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerID="69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f" exitCode=0 Sep 18 18:19:02.302293 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.302218 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" Sep 18 18:19:02.302340 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.302212 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerDied","Data":"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f"} Sep 18 18:19:02.302340 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.302334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c" event={"ID":"bf44764e-5a06-4b8f-b966-d8b3b69389f8","Type":"ContainerDied","Data":"f9e02a00f0a02e6da90fd55e5a65558e698f0685f570464320ce33ea62b0450e"} Sep 18 18:19:02.302431 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.302358 2570 scope.go:117] "RemoveContainer" containerID="69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f" Sep 18 18:19:02.315117 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.315101 2570 scope.go:117] "RemoveContainer" containerID="8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70" Sep 18 18:19:02.323524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.323507 2570 scope.go:117] "RemoveContainer" containerID="69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f" Sep 18 18:19:02.323793 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:02.323776 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f\": container with ID starting with 69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f not found: ID does not exist" containerID="69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f" Sep 18 18:19:02.323860 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.323800 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f"} err="failed to get container status \"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f\": rpc error: code = NotFound desc = could not find container \"69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f\": container with ID starting with 69ec58b155678588f2887e00c453b28edd49215d93fa96c8ae95bdc9da2def7f not found: ID does not exist" Sep 18 18:19:02.323860 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.323814 2570 scope.go:117] "RemoveContainer" containerID="8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70" Sep 18 18:19:02.324054 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:02.324040 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70\": container with ID starting with 8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70 not found: ID does not exist" containerID="8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70" Sep 18 18:19:02.324108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.324058 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70"} err="failed to get container status \"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70\": rpc error: code = NotFound desc = could not find container \"8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70\": container with ID starting with 8dbc6f012ef645888f019ebf1b4d9ea528cab7540b302d5e802fa0d964f92d70 not found: ID does not exist" Sep 18 18:19:02.335139 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.335115 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:19:02.340097 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.340077 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-03e512-openshift-default-6ddd9b7f56-2jn6c"] Sep 18 18:19:02.942140 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:02.942109 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" path="/var/lib/kubelet/pods/bf44764e-5a06-4b8f-b966-d8b3b69389f8/volumes" Sep 18 18:19:09.021924 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.021885 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9"] Sep 18 18:19:09.024191 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.024158 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28ddec-29829255-r6hr9"] Sep 18 18:19:09.048668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.048632 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:19:09.048917 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.048891 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" podUID="849ec89f-477f-4a40-99d9-c977a5e08b80" containerName="payload-processing" containerID="cri-o://dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625" gracePeriod=30 Sep 18 18:19:09.055915 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.055850 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:19:09.056471 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.056100 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" podUID="4c677369-69f1-49ae-8a2a-d6f4067153d1" containerName="payload-pre-processing" containerID="cri-o://5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8" gracePeriod=30 Sep 18 18:19:09.213369 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.213254 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq"] Sep 18 18:19:09.215640 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.215610 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-28ddec-c6whq"] Sep 18 18:19:09.325709 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.325683 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:19:09.328487 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.328471 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:19:09.347762 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.347723 2570 generic.go:356] "Generic (PLEG): container finished" podID="4c677369-69f1-49ae-8a2a-d6f4067153d1" containerID="5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8" exitCode=0 Sep 18 18:19:09.347762 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.347769 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" Sep 18 18:19:09.348008 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.347848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" event={"ID":"4c677369-69f1-49ae-8a2a-d6f4067153d1","Type":"ContainerDied","Data":"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8"} Sep 18 18:19:09.348008 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.347930 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq" event={"ID":"4c677369-69f1-49ae-8a2a-d6f4067153d1","Type":"ContainerDied","Data":"faf44e713e55491b92e0d742ed1760068c506bd71672ef4dad3c5c5c19f6c206"} Sep 18 18:19:09.348008 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.347958 2570 scope.go:117] "RemoveContainer" containerID="5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8" Sep 18 18:19:09.349835 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.349814 2570 generic.go:356] "Generic (PLEG): container finished" podID="849ec89f-477f-4a40-99d9-c977a5e08b80" containerID="dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625" exitCode=0 Sep 18 18:19:09.349932 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.349857 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" Sep 18 18:19:09.349979 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.349938 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" event={"ID":"849ec89f-477f-4a40-99d9-c977a5e08b80","Type":"ContainerDied","Data":"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625"} Sep 18 18:19:09.350019 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.349992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7" event={"ID":"849ec89f-477f-4a40-99d9-c977a5e08b80","Type":"ContainerDied","Data":"cdd48265f65d093a54e6d7a1154d6ad6869b9750435b1739e487e6843df513ca"} Sep 18 18:19:09.357834 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.357813 2570 scope.go:117] "RemoveContainer" containerID="5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8" Sep 18 18:19:09.358246 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:09.358172 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8\": container with ID starting with 5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8 not found: ID does not exist" containerID="5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8" Sep 18 18:19:09.358246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.358222 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8"} err="failed to get container status \"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8\": rpc error: code = NotFound desc = could not find container \"5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8\": container with ID starting with 5f1b3c32962875c059b67a208bdf3551ba1a88f5f750db042dafb1f694db2ac8 not found: ID does not exist" Sep 18 18:19:09.358246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.358247 2570 scope.go:117] "RemoveContainer" containerID="dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625" Sep 18 18:19:09.370861 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.370840 2570 scope.go:117] "RemoveContainer" containerID="dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625" Sep 18 18:19:09.371239 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:09.371221 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625\": container with ID starting with dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625 not found: ID does not exist" containerID="dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625" Sep 18 18:19:09.371312 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.371249 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625"} err="failed to get container status \"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625\": rpc error: code = NotFound desc = could not find container \"dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625\": container with ID starting with dfcbf010f714b9a09f914ac63027de430f28d8bc9f3672e14044979486688625 not found: ID does not exist" Sep 18 18:19:09.391799 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.391765 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c7j5\" (UniqueName: \"kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5\") pod \"849ec89f-477f-4a40-99d9-c977a5e08b80\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " Sep 18 18:19:09.392012 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.391822 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume\") pod \"4c677369-69f1-49ae-8a2a-d6f4067153d1\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " Sep 18 18:19:09.392012 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.391875 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5lxk\" (UniqueName: \"kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk\") pod \"4c677369-69f1-49ae-8a2a-d6f4067153d1\" (UID: \"4c677369-69f1-49ae-8a2a-d6f4067153d1\") " Sep 18 18:19:09.392012 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.391966 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume\") pod \"849ec89f-477f-4a40-99d9-c977a5e08b80\" (UID: \"849ec89f-477f-4a40-99d9-c977a5e08b80\") " Sep 18 18:19:09.392690 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.392656 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "849ec89f-477f-4a40-99d9-c977a5e08b80" (UID: "849ec89f-477f-4a40-99d9-c977a5e08b80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:09.392864 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.392701 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c677369-69f1-49ae-8a2a-d6f4067153d1" (UID: "4c677369-69f1-49ae-8a2a-d6f4067153d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:09.394658 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.394625 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5" (OuterVolumeSpecName: "kube-api-access-7c7j5") pod "849ec89f-477f-4a40-99d9-c977a5e08b80" (UID: "849ec89f-477f-4a40-99d9-c977a5e08b80"). InnerVolumeSpecName "kube-api-access-7c7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:09.394742 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.394670 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk" (OuterVolumeSpecName: "kube-api-access-p5lxk") pod "4c677369-69f1-49ae-8a2a-d6f4067153d1" (UID: "4c677369-69f1-49ae-8a2a-d6f4067153d1"). InnerVolumeSpecName "kube-api-access-p5lxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:09.493703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.493655 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lxk\" (UniqueName: \"kubernetes.io/projected/4c677369-69f1-49ae-8a2a-d6f4067153d1-kube-api-access-p5lxk\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:09.493703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.493694 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/849ec89f-477f-4a40-99d9-c977a5e08b80-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:09.493703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.493704 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7j5\" (UniqueName: \"kubernetes.io/projected/849ec89f-477f-4a40-99d9-c977a5e08b80-kube-api-access-7c7j5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:09.493703 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.493713 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c677369-69f1-49ae-8a2a-d6f4067153d1-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:09.675463 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.675410 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:19:09.678831 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.678802 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28ddec-59d4f5c9c8-qgvqq"] Sep 18 18:19:09.684989 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.684966 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:19:09.690672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:09.690649 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-28ddec-5986dbc4fc-scws7"] Sep 18 18:19:10.942393 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:10.942358 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c677369-69f1-49ae-8a2a-d6f4067153d1" path="/var/lib/kubelet/pods/4c677369-69f1-49ae-8a2a-d6f4067153d1/volumes" Sep 18 18:19:10.942766 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:10.942653 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a948cea-08af-4d0d-a1f4-f03e062df5c0" path="/var/lib/kubelet/pods/6a948cea-08af-4d0d-a1f4-f03e062df5c0/volumes" Sep 18 18:19:10.942912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:10.942899 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa89c94-655c-410e-b46a-b01fa9019561" path="/var/lib/kubelet/pods/7fa89c94-655c-410e-b46a-b01fa9019561/volumes" Sep 18 18:19:10.943235 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:10.943223 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849ec89f-477f-4a40-99d9-c977a5e08b80" path="/var/lib/kubelet/pods/849ec89f-477f-4a40-99d9-c977a5e08b80/volumes" Sep 18 18:19:17.590627 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.590592 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5"] Sep 18 18:19:17.591140 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591119 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c677369-69f1-49ae-8a2a-d6f4067153d1" containerName="payload-pre-processing" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591144 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c677369-69f1-49ae-8a2a-d6f4067153d1" containerName="payload-pre-processing" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591160 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa89c94-655c-410e-b46a-b01fa9019561" containerName="revoke-keys" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591170 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa89c94-655c-410e-b46a-b01fa9019561" containerName="revoke-keys" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591203 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591212 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:19:17.591236 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591231 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849ec89f-477f-4a40-99d9-c977a5e08b80" containerName="payload-processing" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591240 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="849ec89f-477f-4a40-99d9-c977a5e08b80" containerName="payload-processing" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591324 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="849ec89f-477f-4a40-99d9-c977a5e08b80" containerName="payload-processing" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591336 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591342 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2afadaf-500a-4db9-a647-72995df317d8" containerName="istio-proxy" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591347 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591352 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa89c94-655c-410e-b46a-b01fa9019561" containerName="revoke-keys" Sep 18 18:19:17.591552 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.591358 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c677369-69f1-49ae-8a2a-d6f4067153d1" containerName="payload-pre-processing" Sep 18 18:19:17.595769 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.595752 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:17.598743 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.598725 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:19:17.599026 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.599008 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:19:17.607244 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.607218 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5"] Sep 18 18:19:17.669449 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.669409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5\" (UID: \"af60bdf6-bf64-448c-9297-05a4a9b01e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:17.770033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.769995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5\" (UID: \"af60bdf6-bf64-448c-9297-05a4a9b01e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:17.770647 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.770628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5\" (UID: \"af60bdf6-bf64-448c-9297-05a4a9b01e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:17.907664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:17.907630 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:18.186473 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:18.186391 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5"] Sep 18 18:19:18.189447 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:19:18.189421 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf60bdf6_bf64_448c_9297_05a4a9b01e9e.slice/crio-b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788 WatchSource:0}: Error finding container b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788: Status 404 returned error can't find the container with id b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788 Sep 18 18:19:18.404955 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:18.404917 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" event={"ID":"af60bdf6-bf64-448c-9297-05a4a9b01e9e","Type":"ContainerStarted","Data":"b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788"} Sep 18 18:19:19.412293 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:19.412255 2570 generic.go:356] "Generic (PLEG): container finished" podID="af60bdf6-bf64-448c-9297-05a4a9b01e9e" containerID="ebc77113927a856299b9d226d9e8d9154314a2fe5eef2455e04dac18f0ce61e6" exitCode=0 Sep 18 18:19:19.412653 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:19.412346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" event={"ID":"af60bdf6-bf64-448c-9297-05a4a9b01e9e","Type":"ContainerDied","Data":"ebc77113927a856299b9d226d9e8d9154314a2fe5eef2455e04dac18f0ce61e6"} Sep 18 18:19:20.553628 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:20.553603 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:20.595052 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:20.595016 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca\") pod \"af60bdf6-bf64-448c-9297-05a4a9b01e9e\" (UID: \"af60bdf6-bf64-448c-9297-05a4a9b01e9e\") " Sep 18 18:19:20.595375 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:20.595355 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af60bdf6-bf64-448c-9297-05a4a9b01e9e" (UID: "af60bdf6-bf64-448c-9297-05a4a9b01e9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:20.696316 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:20.696233 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af60bdf6-bf64-448c-9297-05a4a9b01e9e-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:21.425220 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:21.425172 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" Sep 18 18:19:21.425395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:21.425162 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5" event={"ID":"af60bdf6-bf64-448c-9297-05a4a9b01e9e","Type":"ContainerDied","Data":"b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788"} Sep 18 18:19:21.425395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:21.425315 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74d024e4e2a636ab8a8f7b33c325571e7d87cfcbeb620cd3ffd5139f2703788" Sep 18 18:19:28.283700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.283669 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr"] Sep 18 18:19:28.288087 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.288057 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-130a80-29829255-rnbkr"] Sep 18 18:19:28.316364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.316337 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:19:28.316560 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.316535 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" podUID="0bcb135a-77cb-42be-8e9b-744930256082" containerName="payload-processing" containerID="cri-o://67702330a681aab975a995f40b1cc916f5357bde974c5dc8f178666e4aa6f3ec" gracePeriod=30 Sep 18 18:19:28.329414 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.329382 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:19:28.329656 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.329629 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" podUID="4fa46635-1848-4b29-bb05-3bda1d446c96" containerName="payload-pre-processing" containerID="cri-o://3ec16b64af4320c5a22c6b3f2d45d7d608e640b1b6058dbe157e578c45c20c2a" gracePeriod=30 Sep 18 18:19:28.477817 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.477776 2570 generic.go:356] "Generic (PLEG): container finished" podID="0bcb135a-77cb-42be-8e9b-744930256082" containerID="67702330a681aab975a995f40b1cc916f5357bde974c5dc8f178666e4aa6f3ec" exitCode=0 Sep 18 18:19:28.477817 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.477815 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" event={"ID":"0bcb135a-77cb-42be-8e9b-744930256082","Type":"ContainerDied","Data":"67702330a681aab975a995f40b1cc916f5357bde974c5dc8f178666e4aa6f3ec"} Sep 18 18:19:28.480227 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.480172 2570 generic.go:356] "Generic (PLEG): container finished" podID="4fa46635-1848-4b29-bb05-3bda1d446c96" containerID="3ec16b64af4320c5a22c6b3f2d45d7d608e640b1b6058dbe157e578c45c20c2a" exitCode=0 Sep 18 18:19:28.480351 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.480252 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" event={"ID":"4fa46635-1848-4b29-bb05-3bda1d446c96","Type":"ContainerDied","Data":"3ec16b64af4320c5a22c6b3f2d45d7d608e640b1b6058dbe157e578c45c20c2a"} Sep 18 18:19:28.519347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.519316 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5"] Sep 18 18:19:28.521707 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.521680 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-130a80-tkvq5"] Sep 18 18:19:28.573858 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.573834 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:19:28.606545 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.606525 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:19:28.665328 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665288 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume\") pod \"4fa46635-1848-4b29-bb05-3bda1d446c96\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " Sep 18 18:19:28.665514 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665352 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6vcr\" (UniqueName: \"kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr\") pod \"4fa46635-1848-4b29-bb05-3bda1d446c96\" (UID: \"4fa46635-1848-4b29-bb05-3bda1d446c96\") " Sep 18 18:19:28.665514 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665391 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume\") pod \"0bcb135a-77cb-42be-8e9b-744930256082\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " Sep 18 18:19:28.665514 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665415 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbwpn\" (UniqueName: \"kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn\") pod \"0bcb135a-77cb-42be-8e9b-744930256082\" (UID: \"0bcb135a-77cb-42be-8e9b-744930256082\") " Sep 18 18:19:28.665785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665736 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fa46635-1848-4b29-bb05-3bda1d446c96" (UID: "4fa46635-1848-4b29-bb05-3bda1d446c96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:28.665785 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.665777 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bcb135a-77cb-42be-8e9b-744930256082" (UID: "0bcb135a-77cb-42be-8e9b-744930256082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:28.667544 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.667521 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr" (OuterVolumeSpecName: "kube-api-access-z6vcr") pod "4fa46635-1848-4b29-bb05-3bda1d446c96" (UID: "4fa46635-1848-4b29-bb05-3bda1d446c96"). InnerVolumeSpecName "kube-api-access-z6vcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:28.667665 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.667644 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn" (OuterVolumeSpecName: "kube-api-access-xbwpn") pod "0bcb135a-77cb-42be-8e9b-744930256082" (UID: "0bcb135a-77cb-42be-8e9b-744930256082"). InnerVolumeSpecName "kube-api-access-xbwpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:28.766993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.766950 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa46635-1848-4b29-bb05-3bda1d446c96-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:28.766993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.766980 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vcr\" (UniqueName: \"kubernetes.io/projected/4fa46635-1848-4b29-bb05-3bda1d446c96-kube-api-access-z6vcr\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:28.766993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.766990 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcb135a-77cb-42be-8e9b-744930256082-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:28.766993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.766999 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbwpn\" (UniqueName: \"kubernetes.io/projected/0bcb135a-77cb-42be-8e9b-744930256082-kube-api-access-xbwpn\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:28.943593 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.943558 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673f3316-4e21-4672-b3c4-6b893a5cd348" path="/var/lib/kubelet/pods/673f3316-4e21-4672-b3c4-6b893a5cd348/volumes" Sep 18 18:19:28.943873 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:28.943860 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af60bdf6-bf64-448c-9297-05a4a9b01e9e" path="/var/lib/kubelet/pods/af60bdf6-bf64-448c-9297-05a4a9b01e9e/volumes" Sep 18 18:19:29.488262 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.488235 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" Sep 18 18:19:29.488262 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.488254 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm" event={"ID":"0bcb135a-77cb-42be-8e9b-744930256082","Type":"ContainerDied","Data":"9f9fa3f3fda2093b077dabe97d109640e87528ffa226eb198f8773b468c149d6"} Sep 18 18:19:29.488781 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.488313 2570 scope.go:117] "RemoveContainer" containerID="67702330a681aab975a995f40b1cc916f5357bde974c5dc8f178666e4aa6f3ec" Sep 18 18:19:29.489819 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.489800 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" Sep 18 18:19:29.489937 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.489840 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8" event={"ID":"4fa46635-1848-4b29-bb05-3bda1d446c96","Type":"ContainerDied","Data":"b08e66e0ba87a58a60d72a0120d11558574c555b463b95c72df2ca8ac36ede83"} Sep 18 18:19:29.497342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.497318 2570 scope.go:117] "RemoveContainer" containerID="3ec16b64af4320c5a22c6b3f2d45d7d608e640b1b6058dbe157e578c45c20c2a" Sep 18 18:19:29.511383 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.511359 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:19:29.514684 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.514660 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-130a80-7448d7fcd6-bjsf8"] Sep 18 18:19:29.527036 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.527015 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:19:29.531929 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:29.531909 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-130a80-8c6495557-9q9wm"] Sep 18 18:19:30.941635 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:30.941605 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bcb135a-77cb-42be-8e9b-744930256082" path="/var/lib/kubelet/pods/0bcb135a-77cb-42be-8e9b-744930256082/volumes" Sep 18 18:19:30.942029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:30.942014 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa46635-1848-4b29-bb05-3bda1d446c96" path="/var/lib/kubelet/pods/4fa46635-1848-4b29-bb05-3bda1d446c96/volumes" Sep 18 18:19:35.022452 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:35.022421 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:19:35.022816 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:35.022608 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" containerID="cri-o://0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd" gracePeriod=30 Sep 18 18:19:37.546639 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:37.546597 2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:19:37.547001 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:37.546676 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:19:40.271073 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.271046 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:19:40.366668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.366668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366630 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.366668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366652 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.366668 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366669 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366697 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh7v7\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366750 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366779 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366801 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo\") pod \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\" (UID: \"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4\") " Sep 18 18:19:40.367010 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.366961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:40.367364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.367086 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:40.367364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.367102 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.367364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.367114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:19:40.367364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.367131 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:40.367364 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.367331 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:40.368936 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.368904 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:40.368936 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.368911 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:19:40.369078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.369006 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7" (OuterVolumeSpecName: "kube-api-access-kh7v7") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "kube-api-access-kh7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:19:40.369078 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.369011 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" (UID: "c9b6d5d2-9894-43ab-82c7-0a92c739c1c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:19:40.468033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.467987 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh7v7\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-kube-api-access-kh7v7\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468023 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468033 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468033 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468046 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468362 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468057 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468362 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468066 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468362 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468074 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.468362 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.468081 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:19:40.553075 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.553043 2570 generic.go:356] "Generic (PLEG): container finished" podID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerID="0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd" exitCode=0 Sep 18 18:19:40.553272 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.553095 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" Sep 18 18:19:40.553272 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.553123 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerDied","Data":"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd"} Sep 18 18:19:40.553272 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.553174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp" event={"ID":"c9b6d5d2-9894-43ab-82c7-0a92c739c1c4","Type":"ContainerDied","Data":"d9977a9053f2dedfe26ca220a282a2c22590d1dad915a1a482beedbbab229234"} Sep 18 18:19:40.553272 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.553211 2570 scope.go:117] "RemoveContainer" containerID="0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd" Sep 18 18:19:40.563857 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.563840 2570 scope.go:117] "RemoveContainer" containerID="e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7" Sep 18 18:19:40.572446 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.572420 2570 scope.go:117] "RemoveContainer" containerID="0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd" Sep 18 18:19:40.572729 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:40.572708 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd\": container with ID starting with 0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd not found: ID does not exist" containerID="0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd" Sep 18 18:19:40.572805 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.572743 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd"} err="failed to get container status \"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd\": rpc error: code = NotFound desc = could not find container \"0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd\": container with ID starting with 0fc0b17a8636d51fbd5d341446cb47d67b61383ff96a1fa2525e7a3032fcf0bd not found: ID does not exist" Sep 18 18:19:40.572805 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.572770 2570 scope.go:117] "RemoveContainer" containerID="e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7" Sep 18 18:19:40.573074 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:19:40.573052 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7\": container with ID starting with e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7 not found: ID does not exist" containerID="e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7" Sep 18 18:19:40.573156 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.573079 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7"} err="failed to get container status \"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7\": rpc error: code = NotFound desc = could not find container \"e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7\": container with ID starting with e35d25544493ab664cd1627a63a4bdd0f3bd14e80d6d8047ef7f842ae4da9da7 not found: ID does not exist" Sep 18 18:19:40.576325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.576299 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:19:40.581256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.581222 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-130a80-openshift-default-fcf59648b-tlxtp"] Sep 18 18:19:40.943146 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:40.943113 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" path="/var/lib/kubelet/pods/c9b6d5d2-9894-43ab-82c7-0a92c739c1c4/volumes" Sep 18 18:19:57.295960 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:19:57.295926 2570 scope.go:117] "RemoveContainer" containerID="1a7900885621c306913a1424935a369a4d58b49b4fe3b8d2e4c26210572a0b81" Sep 18 18:20:55.720053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720019 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720480 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720499 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720515 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720524 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720538 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720547 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf44764e-5a06-4b8f-b966-d8b3b69389f8" containerName="istio-proxy" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720559 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bcb135a-77cb-42be-8e9b-744930256082" containerName="payload-processing" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720568 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bcb135a-77cb-42be-8e9b-744930256082" containerName="payload-processing" Sep 18 18:20:55.720581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720580 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa46635-1848-4b29-bb05-3bda1d446c96" containerName="payload-pre-processing" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720588 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa46635-1848-4b29-bb05-3bda1d446c96" containerName="payload-pre-processing" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720602 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af60bdf6-bf64-448c-9297-05a4a9b01e9e" containerName="revoke-keys" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720609 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="af60bdf6-bf64-448c-9297-05a4a9b01e9e" containerName="revoke-keys" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720690 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bcb135a-77cb-42be-8e9b-744930256082" containerName="payload-processing" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720704 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720714 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="af60bdf6-bf64-448c-9297-05a4a9b01e9e" containerName="revoke-keys" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720724 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa46635-1848-4b29-bb05-3bda1d446c96" containerName="payload-pre-processing" Sep 18 18:20:55.721027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.720735 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b6d5d2-9894-43ab-82c7-0a92c739c1c4" containerName="istio-proxy" Sep 18 18:20:55.723641 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.723622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.727061 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.727038 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"rate-test-model-53cde3-kserve-self-signed-certs\"" Sep 18 18:20:55.727209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.727082 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"default-dockercfg-rks92\"" Sep 18 18:20:55.727209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.727123 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"openshift-service-ca.crt\"" Sep 18 18:20:55.727209 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.727123 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"kube-root-ca.crt\"" Sep 18 18:20:55.732923 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.732896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:20:55.810137 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.810137 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.810365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.810365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.810365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.810365 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.810331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcgd\" (UniqueName: \"kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.911645 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911602 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.911645 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.911879 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.911879 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.911879 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.912028 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.911880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcgd\" (UniqueName: \"kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.912082 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.912046 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.912133 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.912083 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.912215 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.912136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.915398 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.915374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.916091 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.916064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:55.920249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:55.920225 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcgd\" (UniqueName: \"kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd\") pod \"rate-test-model-53cde3-kserve-58f5946955-z6w4b\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:56.034252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:56.034162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:20:56.304308 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:56.304233 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:20:56.307332 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:20:56.307305 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d9b50a7_6504_4783_ab2a_ada28d043e2c.slice/crio-60c465c37aed981d92d25ad20db2604febe37fa2b80c11d64a534975443f5a08 WatchSource:0}: Error finding container 60c465c37aed981d92d25ad20db2604febe37fa2b80c11d64a534975443f5a08: Status 404 returned error can't find the container with id 60c465c37aed981d92d25ad20db2604febe37fa2b80c11d64a534975443f5a08 Sep 18 18:20:56.972835 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:56.972794 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" event={"ID":"5d9b50a7-6504-4783-ab2a-ada28d043e2c","Type":"ContainerStarted","Data":"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840"} Sep 18 18:20:56.973174 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:56.972844 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" event={"ID":"5d9b50a7-6504-4783-ab2a-ada28d043e2c","Type":"ContainerStarted","Data":"60c465c37aed981d92d25ad20db2604febe37fa2b80c11d64a534975443f5a08"} Sep 18 18:20:56.990464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:56.990417 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" podStartSLOduration=1.9904025459999999 podStartE2EDuration="1.990402546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:56.988780106 +0000 UTC m=+1862.081435211" watchObservedRunningTime="2026-09-18 18:20:56.990402546 +0000 UTC m=+1862.083057631" Sep 18 18:20:57.343706 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:20:57.343622 2570 scope.go:117] "RemoveContainer" containerID="412d361566a402f370aee83c54450e12a67b07b34cce724f30011f5cb5ea2181" Sep 18 18:21:06.034866 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:06.034835 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:21:06.035604 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:06.034906 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:21:06.046847 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:06.046816 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:21:07.039840 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:07.039811 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:21:55.942171 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:55.942134 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:21:57.364427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.364392 2570 scope.go:117] "RemoveContainer" containerID="9a39062a7a3f4451da45a7efc457986ec3a378c0ba8edd9c8bcfee1f16b42a75" Sep 18 18:21:57.372739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.372718 2570 scope.go:117] "RemoveContainer" containerID="7849ce34fa1c1fded1003f878d4bc59cbecdccfff056b0e9748e837773465a3e" Sep 18 18:21:57.380343 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.380325 2570 scope.go:117] "RemoveContainer" containerID="c00363f15699e0614c6e5d3cadb61a4442fd22ac5e0b7b36fe59c59817eab27f" Sep 18 18:21:57.665732 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.665694 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:21:57.670579 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.670557 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.673001 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.672982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-66c663-openshift-default-dockercfg-k27bh\"" Sep 18 18:21:57.680694 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.680670 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:21:57.712332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712432 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712475 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.712678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.712534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdv7g\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813049 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813354 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdv7g\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813551 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813771 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.813950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.813930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.815609 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.815589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.815865 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.815846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.821876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.821851 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.822670 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.822645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdv7g\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g\") pod \"e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:57.981874 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:57.981787 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:58.261419 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:58.261387 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:21:58.262447 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:21:58.262420 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb127f8_f370_4bd6_b609_bfa95bde5fcd.slice/crio-e2d37ecd0cfe201f8f2dad942fede7dcd59b65ba94b8d8f0dc10ff675e6da430 WatchSource:0}: Error finding container e2d37ecd0cfe201f8f2dad942fede7dcd59b65ba94b8d8f0dc10ff675e6da430: Status 404 returned error can't find the container with id e2d37ecd0cfe201f8f2dad942fede7dcd59b65ba94b8d8f0dc10ff675e6da430 Sep 18 18:21:58.264758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:58.264723 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:21:58.264848 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:58.264795 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:21:58.264848 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:58.264823 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:21:58.295858 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:58.295834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerStarted","Data":"e2d37ecd0cfe201f8f2dad942fede7dcd59b65ba94b8d8f0dc10ff675e6da430"} Sep 18 18:21:59.302468 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:59.302427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerStarted","Data":"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280"} Sep 18 18:21:59.322891 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:59.322833 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podStartSLOduration=2.322815127 podStartE2EDuration="2.322815127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:21:59.322524325 +0000 UTC m=+1924.415179433" watchObservedRunningTime="2026-09-18 18:21:59.322815127 +0000 UTC m=+1924.415470214" Sep 18 18:21:59.982237 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:59.982200 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:21:59.987214 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:21:59.987173 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:00.307488 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:00.307400 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:00.308833 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:00.308814 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:03.267774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.267741 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:22:03.271246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.271227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.273678 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.273660 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66c663-dockercfg-5zpjl\"" Sep 18 18:22:03.273903 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.273884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66c663\"" Sep 18 18:22:03.283040 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.283018 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:22:03.368243 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.368179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.368537 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.368511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkhj\" (UniqueName: \"kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.469486 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.469454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.469654 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.469498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkhj\" (UniqueName: \"kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.470148 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.470127 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.477194 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.477159 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkhj\" (UniqueName: \"kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj\") pod \"payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.583112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.583024 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:03.853114 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:03.853090 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:22:03.855216 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:22:03.855176 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3e4fe5_dde7_4dd6_bb8c_bd9cd2758ac5.slice/crio-0dc24750953db2ac9b6bd5b033e1d82a7fde9a75706d20c40a156c00a45d06e6 WatchSource:0}: Error finding container 0dc24750953db2ac9b6bd5b033e1d82a7fde9a75706d20c40a156c00a45d06e6: Status 404 returned error can't find the container with id 0dc24750953db2ac9b6bd5b033e1d82a7fde9a75706d20c40a156c00a45d06e6 Sep 18 18:22:04.328448 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:04.328411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" event={"ID":"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5","Type":"ContainerStarted","Data":"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c"} Sep 18 18:22:04.328931 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:04.328467 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" event={"ID":"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5","Type":"ContainerStarted","Data":"0dc24750953db2ac9b6bd5b033e1d82a7fde9a75706d20c40a156c00a45d06e6"} Sep 18 18:22:04.328931 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:04.328488 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:10.962154 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:10.962089 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" podUID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" containerName="main" containerID="cri-o://fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840" gracePeriod=45 Sep 18 18:22:11.254713 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.254681 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:22:11.273446 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.273399 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" podStartSLOduration=8.273382889 podStartE2EDuration="8.273382889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:04.350047256 +0000 UTC m=+1929.442702336" watchObservedRunningTime="2026-09-18 18:22:11.273382889 +0000 UTC m=+1936.366037976" Sep 18 18:22:11.339259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339213 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339428 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339277 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339428 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339303 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339428 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339352 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcgd\" (UniqueName: \"kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339428 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339403 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339428 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339419 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir\") pod \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\" (UID: \"5d9b50a7-6504-4783-ab2a-ada28d043e2c\") " Sep 18 18:22:11.339694 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339512 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:22:11.339694 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339538 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home" (OuterVolumeSpecName: "home") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:22:11.339812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339698 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:22:11.339812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339704 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-home\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:11.339812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.339725 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-model-cache\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:11.341491 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.341468 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm" (OuterVolumeSpecName: "dshm") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:22:11.344171 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.344148 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd" (OuterVolumeSpecName: "kube-api-access-hpcgd") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "kube-api-access-hpcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:22:11.344171 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.344158 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d9b50a7-6504-4783-ab2a-ada28d043e2c" (UID: "5d9b50a7-6504-4783-ab2a-ada28d043e2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 18:22:11.371318 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.371278 2570 generic.go:356] "Generic (PLEG): container finished" podID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" containerID="fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840" exitCode=0 Sep 18 18:22:11.371450 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.371321 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" Sep 18 18:22:11.371450 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.371361 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" event={"ID":"5d9b50a7-6504-4783-ab2a-ada28d043e2c","Type":"ContainerDied","Data":"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840"} Sep 18 18:22:11.371450 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.371403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b" event={"ID":"5d9b50a7-6504-4783-ab2a-ada28d043e2c","Type":"ContainerDied","Data":"60c465c37aed981d92d25ad20db2604febe37fa2b80c11d64a534975443f5a08"} Sep 18 18:22:11.371450 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.371419 2570 scope.go:117] "RemoveContainer" containerID="fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840" Sep 18 18:22:11.380217 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.380197 2570 scope.go:117] "RemoveContainer" containerID="fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840" Sep 18 18:22:11.380566 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:22:11.380547 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840\": container with ID starting with fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840 not found: ID does not exist" containerID="fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840" Sep 18 18:22:11.380684 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.380577 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840"} err="failed to get container status \"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840\": rpc error: code = NotFound desc = could not find container \"fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840\": container with ID starting with fe9fc5cbc9b8b26651bb6adc007c74b4acf1fae1f5bc1d0e83434e880cbce840 not found: ID does not exist" Sep 18 18:22:11.410666 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.410617 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:22:11.414110 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.414081 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/rate-test-model-53cde3-kserve-58f5946955-z6w4b"] Sep 18 18:22:11.440976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.440938 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcgd\" (UniqueName: \"kubernetes.io/projected/5d9b50a7-6504-4783-ab2a-ada28d043e2c-kube-api-access-hpcgd\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:11.440976 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.440970 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tls-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:11.441146 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.441014 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-tmp-dir\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:11.441146 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:11.441028 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9b50a7-6504-4783-ab2a-ada28d043e2c-dshm\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:22:12.942109 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:12.942069 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" path="/var/lib/kubelet/pods/5d9b50a7-6504-4783-ab2a-ada28d043e2c/volumes" Sep 18 18:22:15.308953 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.308919 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:15.309372 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.308982 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:15.337368 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.337339 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:22:15.397608 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.397536 2570 generic.go:356] "Generic (PLEG): container finished" podID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerID="f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280" exitCode=0 Sep 18 18:22:15.397746 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.397605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerDied","Data":"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280"} Sep 18 18:22:15.397965 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.397951 2570 scope.go:117] "RemoveContainer" containerID="f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280" Sep 18 18:22:15.399087 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.399061 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:15.399152 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.399134 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:15.399211 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:15.399163 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:16.410313 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:16.410264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerStarted","Data":"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3"} Sep 18 18:22:16.982492 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:16.982455 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:16.982734 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:16.982713 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:16.982816 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:16.982779 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:17.982366 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:17.982325 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:17.982751 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:17.982400 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:18.982912 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:18.982875 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:18.983323 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:18.982937 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:19.982162 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:19.982117 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:19.982353 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:19.982198 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:20.983081 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:20.983045 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 18 18:22:20.983519 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:20.983098 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 18 18:22:21.986731 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:21.986695 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:21.987130 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:21.987106 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:21.987732 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:21.987714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:22:27.128817 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.128782 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:22:27.129230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.129142 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" containerName="main" Sep 18 18:22:27.129230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.129154 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" containerName="main" Sep 18 18:22:27.129230 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.129226 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9b50a7-6504-4783-ab2a-ada28d043e2c" containerName="main" Sep 18 18:22:27.132322 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.132302 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.135259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.135225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-949a32-openshift-default-dockercfg-ccmht\"" Sep 18 18:22:27.145639 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.145583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:22:27.285454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86mz\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285700 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.285880 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.285730 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86mz\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.386958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.386835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.387297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.387020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.387297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.387087 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.387297 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.387163 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.387464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.387319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.387705 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.387681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.389027 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.389004 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.389259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.389240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.398005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.397985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.398233 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.398215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86mz\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz\") pod \"e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.443440 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.443411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:27.724721 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.724696 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:22:27.730247 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.730179 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:27.730325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.730284 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:27.730325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:27.730313 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:28.482993 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:28.482954 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerStarted","Data":"584b28fa1550b71e4f5fd083955a18c1823fdabfeec403632205e911af61faf6"} Sep 18 18:22:28.483405 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:28.483016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerStarted","Data":"39dfa85a4b31436d2f18df11edb46192a93ba7874373cd1a55f1b2eea5a2d0f9"} Sep 18 18:22:28.503084 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:28.503037 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podStartSLOduration=1.503023105 podStartE2EDuration="1.503023105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:28.501101362 +0000 UTC m=+1953.593756466" watchObservedRunningTime="2026-09-18 18:22:28.503023105 +0000 UTC m=+1953.595678192" Sep 18 18:22:29.443745 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:29.443710 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:29.448719 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:29.448692 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:29.488161 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:29.488135 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:29.488926 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:29.488906 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:33.538107 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.538061 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:22:33.541629 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.541612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.544373 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.544352 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-949a32\"" Sep 18 18:22:33.544578 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.544559 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-949a32-dockercfg-bv5v4\"" Sep 18 18:22:33.551985 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.551963 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:22:33.638911 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.638882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hb4l\" (UniqueName: \"kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.639041 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.638955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.739858 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.739822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hb4l\" (UniqueName: \"kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.740041 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.739885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.740613 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.740594 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.748460 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.748440 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hb4l\" (UniqueName: \"kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l\") pod \"payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:33.853296 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:33.853180 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:34.129622 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:34.129593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:22:34.131234 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:22:34.131200 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888fe30f_90d6_424d_9838_090c0bf5dd34.slice/crio-dbe1aafeda6671af333786e6683acdfb79743f952bb5950c35a1a3cf457279d4 WatchSource:0}: Error finding container dbe1aafeda6671af333786e6683acdfb79743f952bb5950c35a1a3cf457279d4: Status 404 returned error can't find the container with id dbe1aafeda6671af333786e6683acdfb79743f952bb5950c35a1a3cf457279d4 Sep 18 18:22:34.515815 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:34.515780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" event={"ID":"888fe30f-90d6-424d-9838-090c0bf5dd34","Type":"ContainerStarted","Data":"3b72d20dd17914897f0aeba941b24c9c9b382c7d8fdac378152770d3cca2d192"} Sep 18 18:22:34.516071 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:34.515839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" event={"ID":"888fe30f-90d6-424d-9838-090c0bf5dd34","Type":"ContainerStarted","Data":"dbe1aafeda6671af333786e6683acdfb79743f952bb5950c35a1a3cf457279d4"} Sep 18 18:22:34.516071 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:34.515870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:34.535088 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:34.535038 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" podStartSLOduration=1.535022943 podStartE2EDuration="1.535022943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:34.532882921 +0000 UTC m=+1959.625538032" watchObservedRunningTime="2026-09-18 18:22:34.535022943 +0000 UTC m=+1959.627678029" Sep 18 18:22:45.524329 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:45.524235 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:22:48.594715 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.594687 2570 generic.go:356] "Generic (PLEG): container finished" podID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerID="584b28fa1550b71e4f5fd083955a18c1823fdabfeec403632205e911af61faf6" exitCode=0 Sep 18 18:22:48.595153 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.594770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerDied","Data":"584b28fa1550b71e4f5fd083955a18c1823fdabfeec403632205e911af61faf6"} Sep 18 18:22:48.595318 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.595301 2570 scope.go:117] "RemoveContainer" containerID="584b28fa1550b71e4f5fd083955a18c1823fdabfeec403632205e911af61faf6" Sep 18 18:22:48.596384 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.596357 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:48.596470 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.596421 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:48.596470 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:48.596453 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 18:22:49.610855 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:49.610814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerStarted","Data":"d419689cef9eb023bf32eb2cb3d463516fcaf5312fd2b1952c427c1830e5c7de"} Sep 18 18:22:50.443891 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:50.443855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:50.444122 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:50.444100 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 18 18:22:50.444201 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:50.444160 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 18 18:22:51.444259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:51.444219 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 18 18:22:51.444676 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:51.444289 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 18 18:22:52.444705 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:52.444669 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 18 18:22:52.445223 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:52.444724 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 18 18:22:53.444427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:53.444391 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 18 18:22:53.444605 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:53.444451 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 18 18:22:54.444062 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:54.444022 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 18 18:22:54.444432 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:54.444108 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 18 18:22:55.448436 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:55.448406 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:55.448882 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:55.448691 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:22:55.449252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:22:55.449236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:23:43.984094 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:43.984057 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:23:43.987676 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:43.987651 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:43.990317 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:43.990296 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f3eedc-dockercfg-q2zl6\"" Sep 18 18:23:43.990426 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:43.990321 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f3eedc\"" Sep 18 18:23:44.000772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.000745 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:23:44.083977 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.083937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.084315 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.084288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djql5\" (UniqueName: \"kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.185492 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.185460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djql5\" (UniqueName: \"kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.185691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.185560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.186222 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.186202 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.193237 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.193180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djql5\" (UniqueName: \"kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5\") pod \"payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.298108 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.298013 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.569408 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.569318 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:23:44.571670 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:23:44.571636 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135add0e_c90a_4896_915d_af5f8ac7ffa1.slice/crio-9fed9f80aaf7cbf664086348daa9180f36db1a0f6a34251df21736a9269e9072 WatchSource:0}: Error finding container 9fed9f80aaf7cbf664086348daa9180f36db1a0f6a34251df21736a9269e9072: Status 404 returned error can't find the container with id 9fed9f80aaf7cbf664086348daa9180f36db1a0f6a34251df21736a9269e9072 Sep 18 18:23:44.916155 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.916116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" event={"ID":"135add0e-c90a-4896-915d-af5f8ac7ffa1","Type":"ContainerStarted","Data":"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e"} Sep 18 18:23:44.916358 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.916175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" event={"ID":"135add0e-c90a-4896-915d-af5f8ac7ffa1","Type":"ContainerStarted","Data":"9fed9f80aaf7cbf664086348daa9180f36db1a0f6a34251df21736a9269e9072"} Sep 18 18:23:44.916358 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.916229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:23:44.931848 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:44.931793 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" podStartSLOduration=1.9317765740000001 podStartE2EDuration="1.931776574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:23:44.930821541 +0000 UTC m=+2030.023476644" watchObservedRunningTime="2026-09-18 18:23:44.931776574 +0000 UTC m=+2030.024431660" Sep 18 18:23:55.923920 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:23:55.923887 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:24:10.874258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:10.874153 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9"] Sep 18 18:24:10.877099 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:10.877082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:10.879947 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:10.879928 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:24:10.880009 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:10.879952 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:24:10.885025 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:10.885003 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9"] Sep 18 18:24:11.027793 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:11.027761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9\" (UID: \"e5ef0c0d-955a-4967-a0df-f5701c179d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:11.128395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:11.128309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9\" (UID: \"e5ef0c0d-955a-4967-a0df-f5701c179d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:11.128916 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:11.128898 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9\" (UID: \"e5ef0c0d-955a-4967-a0df-f5701c179d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:11.187940 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:11.187915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:11.465485 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:11.461158 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9"] Sep 18 18:24:11.465867 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:24:11.465825 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ef0c0d_955a_4967_a0df_f5701c179d56.slice/crio-5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8 WatchSource:0}: Error finding container 5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8: Status 404 returned error can't find the container with id 5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8 Sep 18 18:24:12.065553 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:12.065466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" event={"ID":"e5ef0c0d-955a-4967-a0df-f5701c179d56","Type":"ContainerStarted","Data":"493cecdac0597cc152c55d41172f6ed9fdf46e1e1cc800ea1532e49c4d4f8c7c"} Sep 18 18:24:12.065553 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:12.065532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" event={"ID":"e5ef0c0d-955a-4967-a0df-f5701c179d56","Type":"ContainerStarted","Data":"5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8"} Sep 18 18:24:12.082289 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:12.082241 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" podStartSLOduration=2.082225597 podStartE2EDuration="2.082225597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:24:12.079907265 +0000 UTC m=+2057.172562372" watchObservedRunningTime="2026-09-18 18:24:12.082225597 +0000 UTC m=+2057.174880726" Sep 18 18:24:13.072395 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:13.072360 2570 generic.go:356] "Generic (PLEG): container finished" podID="e5ef0c0d-955a-4967-a0df-f5701c179d56" containerID="493cecdac0597cc152c55d41172f6ed9fdf46e1e1cc800ea1532e49c4d4f8c7c" exitCode=0 Sep 18 18:24:13.072797 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:13.072442 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" event={"ID":"e5ef0c0d-955a-4967-a0df-f5701c179d56","Type":"ContainerDied","Data":"493cecdac0597cc152c55d41172f6ed9fdf46e1e1cc800ea1532e49c4d4f8c7c"} Sep 18 18:24:14.210237 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:14.210209 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:14.358671 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:14.358581 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca\") pod \"e5ef0c0d-955a-4967-a0df-f5701c179d56\" (UID: \"e5ef0c0d-955a-4967-a0df-f5701c179d56\") " Sep 18 18:24:14.358998 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:14.358971 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5ef0c0d-955a-4967-a0df-f5701c179d56" (UID: "e5ef0c0d-955a-4967-a0df-f5701c179d56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:24:14.459543 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:14.459514 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ef0c0d-955a-4967-a0df-f5701c179d56-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:15.085649 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:15.085622 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" Sep 18 18:24:15.085649 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:15.085637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9" event={"ID":"e5ef0c0d-955a-4967-a0df-f5701c179d56","Type":"ContainerDied","Data":"5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8"} Sep 18 18:24:15.085902 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:15.085688 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff21fd3b186d4508bed2b3100959f425d2c0aaf22fa95bf9d87437da44183f8" Sep 18 18:24:22.143758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.143721 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:24:22.144627 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.144571 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" podUID="135add0e-c90a-4896-915d-af5f8ac7ffa1" containerName="payload-pre-processing" containerID="cri-o://ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e" gracePeriod=30 Sep 18 18:24:22.295235 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.295198 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9"] Sep 18 18:24:22.298120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.298090 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f3eedc-rb6q9"] Sep 18 18:24:22.394020 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.393962 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:24:22.532710 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.532680 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djql5\" (UniqueName: \"kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5\") pod \"135add0e-c90a-4896-915d-af5f8ac7ffa1\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " Sep 18 18:24:22.532873 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.532805 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume\") pod \"135add0e-c90a-4896-915d-af5f8ac7ffa1\" (UID: \"135add0e-c90a-4896-915d-af5f8ac7ffa1\") " Sep 18 18:24:22.533167 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.533142 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "135add0e-c90a-4896-915d-af5f8ac7ffa1" (UID: "135add0e-c90a-4896-915d-af5f8ac7ffa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:24:22.534667 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.534642 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5" (OuterVolumeSpecName: "kube-api-access-djql5") pod "135add0e-c90a-4896-915d-af5f8ac7ffa1" (UID: "135add0e-c90a-4896-915d-af5f8ac7ffa1"). InnerVolumeSpecName "kube-api-access-djql5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:24:22.634258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.634215 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135add0e-c90a-4896-915d-af5f8ac7ffa1-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:22.634258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.634255 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djql5\" (UniqueName: \"kubernetes.io/projected/135add0e-c90a-4896-915d-af5f8ac7ffa1-kube-api-access-djql5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:22.941817 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:22.941778 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ef0c0d-955a-4967-a0df-f5701c179d56" path="/var/lib/kubelet/pods/e5ef0c0d-955a-4967-a0df-f5701c179d56/volumes" Sep 18 18:24:23.132515 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.132481 2570 generic.go:356] "Generic (PLEG): container finished" podID="135add0e-c90a-4896-915d-af5f8ac7ffa1" containerID="ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e" exitCode=0 Sep 18 18:24:23.132690 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.132530 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" Sep 18 18:24:23.132690 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.132548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" event={"ID":"135add0e-c90a-4896-915d-af5f8ac7ffa1","Type":"ContainerDied","Data":"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e"} Sep 18 18:24:23.132690 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.132600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml" event={"ID":"135add0e-c90a-4896-915d-af5f8ac7ffa1","Type":"ContainerDied","Data":"9fed9f80aaf7cbf664086348daa9180f36db1a0f6a34251df21736a9269e9072"} Sep 18 18:24:23.132690 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.132630 2570 scope.go:117] "RemoveContainer" containerID="ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e" Sep 18 18:24:23.141686 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.141668 2570 scope.go:117] "RemoveContainer" containerID="ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e" Sep 18 18:24:23.141950 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:24:23.141921 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e\": container with ID starting with ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e not found: ID does not exist" containerID="ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e" Sep 18 18:24:23.142006 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.141950 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e"} err="failed to get container status \"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e\": rpc error: code = NotFound desc = could not find container \"ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e\": container with ID starting with ccebbc08476ecb57d7f00eb5999eb0ee72d5447da4f107635f033cb6c338214e not found: ID does not exist" Sep 18 18:24:23.147410 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.147383 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:24:23.151550 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:23.151527 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f3eedc-5749cb48c8-lfcml"] Sep 18 18:24:24.941531 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:24.941496 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135add0e-c90a-4896-915d-af5f8ac7ffa1" path="/var/lib/kubelet/pods/135add0e-c90a-4896-915d-af5f8ac7ffa1/volumes" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.294662 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4"] Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295509 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ef0c0d-955a-4967-a0df-f5701c179d56" containerName="revoke-keys" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295532 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ef0c0d-955a-4967-a0df-f5701c179d56" containerName="revoke-keys" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295567 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135add0e-c90a-4896-915d-af5f8ac7ffa1" containerName="payload-pre-processing" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295577 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="135add0e-c90a-4896-915d-af5f8ac7ffa1" containerName="payload-pre-processing" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295723 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ef0c0d-955a-4967-a0df-f5701c179d56" containerName="revoke-keys" Sep 18 18:24:32.297363 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.295736 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="135add0e-c90a-4896-915d-af5f8ac7ffa1" containerName="payload-pre-processing" Sep 18 18:24:32.301013 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.300986 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:32.303841 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.303817 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:24:32.304021 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.303853 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:24:32.304454 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.304428 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4"] Sep 18 18:24:32.323094 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.323069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4\" (UID: \"6ab19fa8-924a-4b65-a624-ae5f3256e669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:32.423541 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.423516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4\" (UID: \"6ab19fa8-924a-4b65-a624-ae5f3256e669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:32.424105 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.424088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4\" (UID: \"6ab19fa8-924a-4b65-a624-ae5f3256e669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:32.612551 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.612477 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:32.885612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:32.885588 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4"] Sep 18 18:24:32.887314 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:24:32.887283 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab19fa8_924a_4b65_a624_ae5f3256e669.slice/crio-f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6 WatchSource:0}: Error finding container f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6: Status 404 returned error can't find the container with id f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6 Sep 18 18:24:33.194943 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:33.194868 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" event={"ID":"6ab19fa8-924a-4b65-a624-ae5f3256e669","Type":"ContainerStarted","Data":"f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6"} Sep 18 18:24:34.201813 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:34.201782 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ab19fa8-924a-4b65-a624-ae5f3256e669" containerID="26e46f2dcd535111278faba4f063d6c5ff6a68fae047cb98c9dfd7bdf8c0613f" exitCode=0 Sep 18 18:24:34.202206 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:34.201917 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" event={"ID":"6ab19fa8-924a-4b65-a624-ae5f3256e669","Type":"ContainerDied","Data":"26e46f2dcd535111278faba4f063d6c5ff6a68fae047cb98c9dfd7bdf8c0613f"} Sep 18 18:24:35.339725 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:35.339700 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:35.448651 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:35.448612 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca\") pod \"6ab19fa8-924a-4b65-a624-ae5f3256e669\" (UID: \"6ab19fa8-924a-4b65-a624-ae5f3256e669\") " Sep 18 18:24:35.448950 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:35.448930 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ab19fa8-924a-4b65-a624-ae5f3256e669" (UID: "6ab19fa8-924a-4b65-a624-ae5f3256e669"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:24:35.550120 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:35.550045 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab19fa8-924a-4b65-a624-ae5f3256e669-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:36.214899 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:36.214866 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" Sep 18 18:24:36.214899 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:36.214893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4" event={"ID":"6ab19fa8-924a-4b65-a624-ae5f3256e669","Type":"ContainerDied","Data":"f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6"} Sep 18 18:24:36.215106 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:36.214933 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6668426c25df160619ce3b7dd125ad5b57649ea192acf465b915aad09fdfdf6" Sep 18 18:24:47.874783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:47.874751 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:24:47.875384 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:47.875009 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" podUID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" containerName="payload-pre-processing" containerID="cri-o://6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c" gracePeriod=30 Sep 18 18:24:48.029355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.029312 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4"] Sep 18 18:24:48.031812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.031780 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-66c663-kr9x4"] Sep 18 18:24:48.141998 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.141934 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:24:48.261163 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.261133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btkhj\" (UniqueName: \"kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj\") pod \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " Sep 18 18:24:48.261379 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.261221 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume\") pod \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\" (UID: \"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5\") " Sep 18 18:24:48.261577 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.261554 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" (UID: "ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:24:48.263300 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.263279 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj" (OuterVolumeSpecName: "kube-api-access-btkhj") pod "ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" (UID: "ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5"). InnerVolumeSpecName "kube-api-access-btkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:24:48.281962 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.281931 2570 generic.go:356] "Generic (PLEG): container finished" podID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" containerID="6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c" exitCode=0 Sep 18 18:24:48.282053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.281990 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" Sep 18 18:24:48.282053 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.282013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" event={"ID":"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5","Type":"ContainerDied","Data":"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c"} Sep 18 18:24:48.282126 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.282064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm" event={"ID":"ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5","Type":"ContainerDied","Data":"0dc24750953db2ac9b6bd5b033e1d82a7fde9a75706d20c40a156c00a45d06e6"} Sep 18 18:24:48.282126 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.282087 2570 scope.go:117] "RemoveContainer" containerID="6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c" Sep 18 18:24:48.291250 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.291233 2570 scope.go:117] "RemoveContainer" containerID="6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c" Sep 18 18:24:48.291498 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:24:48.291481 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c\": container with ID starting with 6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c not found: ID does not exist" containerID="6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c" Sep 18 18:24:48.291557 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.291504 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c"} err="failed to get container status \"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c\": rpc error: code = NotFound desc = could not find container \"6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c\": container with ID starting with 6491ec428f227fc521d788b1cd5d4f985c7c48e2e90ce78713c4ebc286ab0d6c not found: ID does not exist" Sep 18 18:24:48.317342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.317301 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:24:48.336567 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.334271 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66c663-5fc87fdc66-7grlm"] Sep 18 18:24:48.362754 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.362717 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkhj\" (UniqueName: \"kubernetes.io/projected/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-kube-api-access-btkhj\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:48.362754 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.362750 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:24:48.941561 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.941521 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab19fa8-924a-4b65-a624-ae5f3256e669" path="/var/lib/kubelet/pods/6ab19fa8-924a-4b65-a624-ae5f3256e669/volumes" Sep 18 18:24:48.941934 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:48.941900 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" path="/var/lib/kubelet/pods/ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5/volumes" Sep 18 18:24:57.429773 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:57.429740 2570 scope.go:117] "RemoveContainer" containerID="a04ec5738d1d71c8f17bc3438a78a7c5b8d6cdb47e37c1a71551a6387a30bd57" Sep 18 18:24:57.438776 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:57.438754 2570 scope.go:117] "RemoveContainer" containerID="d7a76780d0cfb2ae101adab71f47c13883bb423b43d6513b3ff15dc2b2ac6a26" Sep 18 18:24:57.446826 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:24:57.446804 2570 scope.go:117] "RemoveContainer" containerID="c8874f2b149fc98abe95778dbaf168560d6226743202eb87aae84d9f083e67a6" Sep 18 18:25:36.264812 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.264775 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:25:36.265261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.264986 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" containerID="cri-o://49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3" gracePeriod=30 Sep 18 18:25:36.651234 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651129 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj"] Sep 18 18:25:36.651576 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651557 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab19fa8-924a-4b65-a624-ae5f3256e669" containerName="revoke-keys" Sep 18 18:25:36.651664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651578 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab19fa8-924a-4b65-a624-ae5f3256e669" containerName="revoke-keys" Sep 18 18:25:36.651664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651598 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" containerName="payload-pre-processing" Sep 18 18:25:36.651664 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651607 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" containerName="payload-pre-processing" Sep 18 18:25:36.651824 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651698 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3e4fe5-dde7-4dd6-bb8c-bd9cd2758ac5" containerName="payload-pre-processing" Sep 18 18:25:36.651824 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.651711 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab19fa8-924a-4b65-a624-ae5f3256e669" containerName="revoke-keys" Sep 18 18:25:36.654508 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.654486 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:36.656933 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.656911 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:25:36.657030 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.656911 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:25:36.661177 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.661155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj"] Sep 18 18:25:36.673768 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.673747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj\" (UID: \"a760c189-b67d-4a3b-8440-ab869953549a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:36.774834 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.774801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj\" (UID: \"a760c189-b67d-4a3b-8440-ab869953549a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:36.775581 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.775563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj\" (UID: \"a760c189-b67d-4a3b-8440-ab869953549a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:36.965261 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.965229 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:36.989349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.989316 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:25:36.989468 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:36.989387 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:25:37.241603 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:37.241576 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj"] Sep 18 18:25:37.243558 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:25:37.243531 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda760c189_b67d_4a3b_8440_ab869953549a.slice/crio-2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7 WatchSource:0}: Error finding container 2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7: Status 404 returned error can't find the container with id 2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7 Sep 18 18:25:37.553909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:37.553877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" event={"ID":"a760c189-b67d-4a3b-8440-ab869953549a","Type":"ContainerStarted","Data":"45be18817f500a2b4b33f70d7f87381d2ceb7d6f67b39fcb53402f2bc5e60b11"} Sep 18 18:25:37.554274 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:37.553954 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" event={"ID":"a760c189-b67d-4a3b-8440-ab869953549a","Type":"ContainerStarted","Data":"2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7"} Sep 18 18:25:37.570168 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:37.570119 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" podStartSLOduration=1.570104554 podStartE2EDuration="1.570104554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:25:37.568287935 +0000 UTC m=+2142.660943028" watchObservedRunningTime="2026-09-18 18:25:37.570104554 +0000 UTC m=+2142.662759642" Sep 18 18:25:38.560811 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:38.560777 2570 generic.go:356] "Generic (PLEG): container finished" podID="a760c189-b67d-4a3b-8440-ab869953549a" containerID="45be18817f500a2b4b33f70d7f87381d2ceb7d6f67b39fcb53402f2bc5e60b11" exitCode=0 Sep 18 18:25:38.561225 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:38.560865 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" event={"ID":"a760c189-b67d-4a3b-8440-ab869953549a","Type":"ContainerDied","Data":"45be18817f500a2b4b33f70d7f87381d2ceb7d6f67b39fcb53402f2bc5e60b11"} Sep 18 18:25:39.692355 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:39.692331 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:39.802255 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:39.802210 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca\") pod \"a760c189-b67d-4a3b-8440-ab869953549a\" (UID: \"a760c189-b67d-4a3b-8440-ab869953549a\") " Sep 18 18:25:39.802558 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:39.802536 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a760c189-b67d-4a3b-8440-ab869953549a" (UID: "a760c189-b67d-4a3b-8440-ab869953549a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:25:39.903866 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:39.903829 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a760c189-b67d-4a3b-8440-ab869953549a-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:40.574238 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:40.574209 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" Sep 18 18:25:40.574509 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:40.574213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj" event={"ID":"a760c189-b67d-4a3b-8440-ab869953549a","Type":"ContainerDied","Data":"2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7"} Sep 18 18:25:40.574509 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:40.574336 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2355a6d0021e887236b3379bf22e9a46d24ab21d9ed66761252e580fd4c9a9b7" Sep 18 18:25:41.514342 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.514317 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:25:41.581242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.581158 2570 generic.go:356] "Generic (PLEG): container finished" podID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerID="49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3" exitCode=0 Sep 18 18:25:41.581242 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.581235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerDied","Data":"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3"} Sep 18 18:25:41.581435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.581279 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" event={"ID":"abb127f8-f370-4bd6-b609-bfa95bde5fcd","Type":"ContainerDied","Data":"e2d37ecd0cfe201f8f2dad942fede7dcd59b65ba94b8d8f0dc10ff675e6da430"} Sep 18 18:25:41.581435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.581302 2570 scope.go:117] "RemoveContainer" containerID="49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3" Sep 18 18:25:41.581435 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.581241 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx" Sep 18 18:25:41.590163 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.590144 2570 scope.go:117] "RemoveContainer" containerID="f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280" Sep 18 18:25:41.597994 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.597970 2570 scope.go:117] "RemoveContainer" containerID="49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3" Sep 18 18:25:41.598301 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:25:41.598277 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3\": container with ID starting with 49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3 not found: ID does not exist" containerID="49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3" Sep 18 18:25:41.598399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.598311 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3"} err="failed to get container status \"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3\": rpc error: code = NotFound desc = could not find container \"49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3\": container with ID starting with 49b08c84185b66b160adb90db12d5cabe5ff4d618bc61c3a82a0320ce7c2e6f3 not found: ID does not exist" Sep 18 18:25:41.598399 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.598335 2570 scope.go:117] "RemoveContainer" containerID="f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280" Sep 18 18:25:41.598597 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:25:41.598580 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280\": container with ID starting with f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280 not found: ID does not exist" containerID="f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280" Sep 18 18:25:41.598644 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.598606 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280"} err="failed to get container status \"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280\": rpc error: code = NotFound desc = could not find container \"f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280\": container with ID starting with f917632b6508d0997a96f03b39fe0e039488ce15c733caf462d2c39fb384b280 not found: ID does not exist" Sep 18 18:25:41.616901 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.616877 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdv7g\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.616908 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.616944 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.616971 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617005 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.616994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617051 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617082 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617117 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket\") pod \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\" (UID: \"abb127f8-f370-4bd6-b609-bfa95bde5fcd\") " Sep 18 18:25:41.617256 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617157 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:41.617587 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617480 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:41.617587 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617488 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.617587 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617516 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:25:41.617748 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617607 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:41.617748 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.617662 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:41.619171 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.619145 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g" (OuterVolumeSpecName: "kube-api-access-pdv7g") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "kube-api-access-pdv7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:25:41.619282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.619213 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:25:41.619498 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.619471 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:41.619540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.619525 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abb127f8-f370-4bd6-b609-bfa95bde5fcd" (UID: "abb127f8-f370-4bd6-b609-bfa95bde5fcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:25:41.718110 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718082 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdv7g\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-kube-api-access-pdv7g\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718110 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718106 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718110 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718115 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718123 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718132 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718139 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718146 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb127f8-f370-4bd6-b609-bfa95bde5fcd-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.718332 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.718154 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb127f8-f370-4bd6-b609-bfa95bde5fcd-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:41.907107 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.907079 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:25:41.912480 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:41.912449 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-66c663-openshift-default-66fd9cb998-zrvmx"] Sep 18 18:25:42.941651 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:42.941618 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" path="/var/lib/kubelet/pods/abb127f8-f370-4bd6-b609-bfa95bde5fcd/volumes" Sep 18 18:25:47.427762 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.427725 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:25:47.428239 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.427978 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" podUID="888fe30f-90d6-424d-9838-090c0bf5dd34" containerName="payload-pre-processing" containerID="cri-o://3b72d20dd17914897f0aeba941b24c9c9b382c7d8fdac378152770d3cca2d192" gracePeriod=30 Sep 18 18:25:47.616911 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.616877 2570 generic.go:356] "Generic (PLEG): container finished" podID="888fe30f-90d6-424d-9838-090c0bf5dd34" containerID="3b72d20dd17914897f0aeba941b24c9c9b382c7d8fdac378152770d3cca2d192" exitCode=0 Sep 18 18:25:47.617048 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.616968 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" event={"ID":"888fe30f-90d6-424d-9838-090c0bf5dd34","Type":"ContainerDied","Data":"3b72d20dd17914897f0aeba941b24c9c9b382c7d8fdac378152770d3cca2d192"} Sep 18 18:25:47.626258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.626228 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj"] Sep 18 18:25:47.629294 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.629272 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-949a32-bsscj"] Sep 18 18:25:47.677089 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.677068 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:25:47.768211 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.768110 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hb4l\" (UniqueName: \"kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l\") pod \"888fe30f-90d6-424d-9838-090c0bf5dd34\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " Sep 18 18:25:47.768211 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.768169 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume\") pod \"888fe30f-90d6-424d-9838-090c0bf5dd34\" (UID: \"888fe30f-90d6-424d-9838-090c0bf5dd34\") " Sep 18 18:25:47.768529 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.768505 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "888fe30f-90d6-424d-9838-090c0bf5dd34" (UID: "888fe30f-90d6-424d-9838-090c0bf5dd34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:25:47.770258 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.770237 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l" (OuterVolumeSpecName: "kube-api-access-9hb4l") pod "888fe30f-90d6-424d-9838-090c0bf5dd34" (UID: "888fe30f-90d6-424d-9838-090c0bf5dd34"). InnerVolumeSpecName "kube-api-access-9hb4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:25:47.869349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.869316 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hb4l\" (UniqueName: \"kubernetes.io/projected/888fe30f-90d6-424d-9838-090c0bf5dd34-kube-api-access-9hb4l\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:47.869349 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:47.869343 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888fe30f-90d6-424d-9838-090c0bf5dd34-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:48.622887 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.622852 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" event={"ID":"888fe30f-90d6-424d-9838-090c0bf5dd34","Type":"ContainerDied","Data":"dbe1aafeda6671af333786e6683acdfb79743f952bb5950c35a1a3cf457279d4"} Sep 18 18:25:48.623325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.622909 2570 scope.go:117] "RemoveContainer" containerID="3b72d20dd17914897f0aeba941b24c9c9b382c7d8fdac378152770d3cca2d192" Sep 18 18:25:48.623325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.622864 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv" Sep 18 18:25:48.655124 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.655093 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:25:48.659302 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.659277 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-949a32-86bc4857c4-4jfpv"] Sep 18 18:25:48.941907 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.941875 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888fe30f-90d6-424d-9838-090c0bf5dd34" path="/var/lib/kubelet/pods/888fe30f-90d6-424d-9838-090c0bf5dd34/volumes" Sep 18 18:25:48.942265 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:48.942251 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a760c189-b67d-4a3b-8440-ab869953549a" path="/var/lib/kubelet/pods/a760c189-b67d-4a3b-8440-ab869953549a/volumes" Sep 18 18:25:54.508982 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.508950 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:25:54.509458 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.509215 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" containerID="cri-o://d419689cef9eb023bf32eb2cb3d463516fcaf5312fd2b1952c427c1830e5c7de" gracePeriod=30 Sep 18 18:25:54.875522 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875437 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs"] Sep 18 18:25:54.875769 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875756 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888fe30f-90d6-424d-9838-090c0bf5dd34" containerName="payload-pre-processing" Sep 18 18:25:54.875821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875770 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="888fe30f-90d6-424d-9838-090c0bf5dd34" containerName="payload-pre-processing" Sep 18 18:25:54.875821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875784 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a760c189-b67d-4a3b-8440-ab869953549a" containerName="revoke-keys" Sep 18 18:25:54.875821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875790 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a760c189-b67d-4a3b-8440-ab869953549a" containerName="revoke-keys" Sep 18 18:25:54.875821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875803 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:25:54.875821 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875810 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:25:54.875975 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875862 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:25:54.875975 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875870 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:25:54.875975 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875877 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="888fe30f-90d6-424d-9838-090c0bf5dd34" containerName="payload-pre-processing" Sep 18 18:25:54.875975 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.875884 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a760c189-b67d-4a3b-8440-ab869953549a" containerName="revoke-keys" Sep 18 18:25:54.878544 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.878526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:54.880878 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.880854 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:25:54.880989 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.880925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:25:54.887248 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.887224 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs"] Sep 18 18:25:54.926065 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:54.926039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs\" (UID: \"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:55.027025 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.026994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs\" (UID: \"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:55.027612 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.027593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs\" (UID: \"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:55.189903 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.189867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:55.451547 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.451473 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 18:25:55.451680 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.451541 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 18:25:55.461054 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.461028 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs"] Sep 18 18:25:55.462273 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:25:55.462251 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dc37ae_3ca8_4f1c_98d4_762a14f162c7.slice/crio-9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970 WatchSource:0}: Error finding container 9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970: Status 404 returned error can't find the container with id 9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970 Sep 18 18:25:55.668136 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:55.668097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" event={"ID":"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7","Type":"ContainerStarted","Data":"9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970"} Sep 18 18:25:56.674725 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:56.674683 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" containerID="2eab8c79b2fd71e5a166dea2865178ee28d07195c47ea18943588e78121317bb" exitCode=0 Sep 18 18:25:56.675167 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:56.674768 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" event={"ID":"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7","Type":"ContainerDied","Data":"2eab8c79b2fd71e5a166dea2865178ee28d07195c47ea18943588e78121317bb"} Sep 18 18:25:57.481068 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:57.481037 2570 scope.go:117] "RemoveContainer" containerID="ebc77113927a856299b9d226d9e8d9154314a2fe5eef2455e04dac18f0ce61e6" Sep 18 18:25:57.489310 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:57.489292 2570 scope.go:117] "RemoveContainer" containerID="2d4d95039847e05c65db504081ec06a4001e2e54af0c4dc07c1ef1b8f69fa62b" Sep 18 18:25:57.809898 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:57.809875 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:57.952572 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:57.952544 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca\") pod \"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7\" (UID: \"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7\") " Sep 18 18:25:57.952929 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:57.952911 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" (UID: "d9dc37ae-3ca8-4f1c-98d4-762a14f162c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:25:58.053504 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:58.053423 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:58.687154 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:58.687127 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" Sep 18 18:25:58.687334 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:58.687126 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs" event={"ID":"d9dc37ae-3ca8-4f1c-98d4-762a14f162c7","Type":"ContainerDied","Data":"9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970"} Sep 18 18:25:58.687334 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:58.687251 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcf43d15909f7d24d84c403af0664b7fb8c7d47ee72a5df38be706e3a7bf970" Sep 18 18:25:59.695829 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.695796 2570 generic.go:356] "Generic (PLEG): container finished" podID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerID="d419689cef9eb023bf32eb2cb3d463516fcaf5312fd2b1952c427c1830e5c7de" exitCode=0 Sep 18 18:25:59.696429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.695869 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerDied","Data":"d419689cef9eb023bf32eb2cb3d463516fcaf5312fd2b1952c427c1830e5c7de"} Sep 18 18:25:59.696429 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.695938 2570 scope.go:117] "RemoveContainer" containerID="584b28fa1550b71e4f5fd083955a18c1823fdabfeec403632205e911af61faf6" Sep 18 18:25:59.761299 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.761272 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:25:59.872604 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872494 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872604 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872569 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872604 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872603 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872893 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872654 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872893 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872690 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h86mz\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872893 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872734 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.872893 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872761 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872911 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873023 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.872984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873040 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data" (OuterVolumeSpecName: "istio-data") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873053 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:59.873096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873086 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy\") pod \"82377d21-ca60-4c5d-88cc-a497d7a74702\" (UID: \"82377d21-ca60-4c5d-88cc-a497d7a74702\") " Sep 18 18:25:59.873386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873150 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:59.873451 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873427 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-certs\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.873505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873451 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-credential-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.873505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873468 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-workload-socket\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.873505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873481 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82377d21-ca60-4c5d-88cc-a497d7a74702-istiod-ca-cert\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.873505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.873494 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-data\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.874941 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.874907 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token" (OuterVolumeSpecName: "istio-token") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:25:59.875069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.874959 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 18:25:59.875069 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.874993 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 18:25:59.875215 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.875069 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz" (OuterVolumeSpecName: "kube-api-access-h86mz") pod "82377d21-ca60-4c5d-88cc-a497d7a74702" (UID: "82377d21-ca60-4c5d-88cc-a497d7a74702"). InnerVolumeSpecName "kube-api-access-h86mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:25:59.974118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.974068 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86mz\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-kube-api-access-h86mz\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.974118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.974112 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-envoy\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.974118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.974121 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-podinfo\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:25:59.974118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:25:59.974129 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82377d21-ca60-4c5d-88cc-a497d7a74702-istio-token\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:00.702196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.702167 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" Sep 18 18:26:00.702652 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.702195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx" event={"ID":"82377d21-ca60-4c5d-88cc-a497d7a74702","Type":"ContainerDied","Data":"39dfa85a4b31436d2f18df11edb46192a93ba7874373cd1a55f1b2eea5a2d0f9"} Sep 18 18:26:00.702652 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.702251 2570 scope.go:117] "RemoveContainer" containerID="d419689cef9eb023bf32eb2cb3d463516fcaf5312fd2b1952c427c1830e5c7de" Sep 18 18:26:00.729347 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.729312 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:26:00.734436 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.734410 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-949a32-openshift-default-cb5548448-ls4qx"] Sep 18 18:26:00.941409 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:00.941374 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" path="/var/lib/kubelet/pods/82377d21-ca60-4c5d-88cc-a497d7a74702/volumes" Sep 18 18:26:05.373782 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.373748 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:26:05.374296 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.373977 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" podUID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" containerName="payload-pre-processing" containerID="cri-o://225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74" gracePeriod=30 Sep 18 18:26:05.530227 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.530167 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs"] Sep 18 18:26:05.533455 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.533424 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-53cde3-g8bhs"] Sep 18 18:26:05.633143 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.633073 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:26:05.723595 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.723567 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpgl5\" (UniqueName: \"kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5\") pod \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " Sep 18 18:26:05.723760 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.723715 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume\") pod \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\" (UID: \"b3161b5f-3ba6-4686-b18e-6d286e0ff09f\") " Sep 18 18:26:05.724068 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.724040 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3161b5f-3ba6-4686-b18e-6d286e0ff09f" (UID: "b3161b5f-3ba6-4686-b18e-6d286e0ff09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:26:05.725851 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.725824 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5" (OuterVolumeSpecName: "kube-api-access-rpgl5") pod "b3161b5f-3ba6-4686-b18e-6d286e0ff09f" (UID: "b3161b5f-3ba6-4686-b18e-6d286e0ff09f"). InnerVolumeSpecName "kube-api-access-rpgl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:26:05.729611 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.729586 2570 generic.go:356] "Generic (PLEG): container finished" podID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" containerID="225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74" exitCode=0 Sep 18 18:26:05.729698 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.729625 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" Sep 18 18:26:05.729698 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.729672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" event={"ID":"b3161b5f-3ba6-4686-b18e-6d286e0ff09f","Type":"ContainerDied","Data":"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74"} Sep 18 18:26:05.729774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.729724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54" event={"ID":"b3161b5f-3ba6-4686-b18e-6d286e0ff09f","Type":"ContainerDied","Data":"1545ea571d2aad91fdb4ed84d76ffd692ba1c5f2d16e76aeee349342b113f2e4"} Sep 18 18:26:05.729774 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.729751 2570 scope.go:117] "RemoveContainer" containerID="225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74" Sep 18 18:26:05.739295 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.739275 2570 scope.go:117] "RemoveContainer" containerID="225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74" Sep 18 18:26:05.739611 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:26:05.739581 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74\": container with ID starting with 225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74 not found: ID does not exist" containerID="225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74" Sep 18 18:26:05.739685 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.739619 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74"} err="failed to get container status \"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74\": rpc error: code = NotFound desc = could not find container \"225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74\": container with ID starting with 225c0a23e79759b555c694aaf54a5733c27d7afb2444c8db3bb6243995a34b74 not found: ID does not exist" Sep 18 18:26:05.751536 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.751513 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:26:05.754527 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.754503 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-53cde3-597b67bf46-9kl54"] Sep 18 18:26:05.824636 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.824606 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:05.824636 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:05.824631 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpgl5\" (UniqueName: \"kubernetes.io/projected/b3161b5f-3ba6-4686-b18e-6d286e0ff09f-kube-api-access-rpgl5\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:06.941758 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:06.941723 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" path="/var/lib/kubelet/pods/b3161b5f-3ba6-4686-b18e-6d286e0ff09f/volumes" Sep 18 18:26:06.942136 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:06.942067 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" path="/var/lib/kubelet/pods/d9dc37ae-3ca8-4f1c-98d4-762a14f162c7/volumes" Sep 18 18:26:12.430512 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430474 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f"] Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430783 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430793 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430807 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" containerName="revoke-keys" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430812 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" containerName="revoke-keys" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430821 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430827 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430839 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" containerName="payload-pre-processing" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430845 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" containerName="payload-pre-processing" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430851 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430857 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb127f8-f370-4bd6-b609-bfa95bde5fcd" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430921 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3161b5f-3ba6-4686-b18e-6d286e0ff09f" containerName="payload-pre-processing" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430928 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430936 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dc37ae-3ca8-4f1c-98d4-762a14f162c7" containerName="revoke-keys" Sep 18 18:26:12.430970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.430943 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82377d21-ca60-4c5d-88cc-a497d7a74702" containerName="istio-proxy" Sep 18 18:26:12.434876 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.434859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:12.437116 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.437085 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 18:26:12.437252 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.437098 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88z7v\"" Sep 18 18:26:12.441781 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.441755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f"] Sep 18 18:26:12.583052 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.583017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f\" (UID: \"f8e74640-2460-4173-9d97-ee0f94790872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:12.683737 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.683657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f\" (UID: \"f8e74640-2460-4173-9d97-ee0f94790872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:12.684333 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.684313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f\" (UID: \"f8e74640-2460-4173-9d97-ee0f94790872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:12.745492 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:12.745457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:13.013096 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:13.013065 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f"] Sep 18 18:26:13.015003 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:26:13.014979 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e74640_2460_4173_9d97_ee0f94790872.slice/crio-dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3 WatchSource:0}: Error finding container dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3: Status 404 returned error can't find the container with id dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3 Sep 18 18:26:13.777346 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:13.777315 2570 generic.go:356] "Generic (PLEG): container finished" podID="f8e74640-2460-4173-9d97-ee0f94790872" containerID="391be43e6581e47c9c2ec9d39d6fb4f82feefb3714a9fb3d05248b0805255deb" exitCode=0 Sep 18 18:26:13.777707 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:13.777377 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" event={"ID":"f8e74640-2460-4173-9d97-ee0f94790872","Type":"ContainerDied","Data":"391be43e6581e47c9c2ec9d39d6fb4f82feefb3714a9fb3d05248b0805255deb"} Sep 18 18:26:13.777707 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:13.777433 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" event={"ID":"f8e74640-2460-4173-9d97-ee0f94790872","Type":"ContainerStarted","Data":"dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3"} Sep 18 18:26:14.908720 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:14.908693 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:15.004326 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.004299 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca\") pod \"f8e74640-2460-4173-9d97-ee0f94790872\" (UID: \"f8e74640-2460-4173-9d97-ee0f94790872\") " Sep 18 18:26:15.004646 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.004622 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e74640-2460-4173-9d97-ee0f94790872" (UID: "f8e74640-2460-4173-9d97-ee0f94790872"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:26:15.105179 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.105101 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e74640-2460-4173-9d97-ee0f94790872-maas-api-service-ca\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:15.790128 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.790100 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" Sep 18 18:26:15.790325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.790093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f" event={"ID":"f8e74640-2460-4173-9d97-ee0f94790872","Type":"ContainerDied","Data":"dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3"} Sep 18 18:26:15.790325 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:15.790250 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc52e840c98a9cb315972839d9d9c0a81e91fcc8c3337181207aff0bd6c978e3" Sep 18 18:26:22.865512 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.865472 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt"] Sep 18 18:26:22.868409 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.868375 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-341bd4-29829255-qh9gt"] Sep 18 18:26:22.889968 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.889937 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:26:22.890241 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.890172 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" podUID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" containerName="payload-processing" containerID="cri-o://b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13" gracePeriod=30 Sep 18 18:26:22.896684 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.896656 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:26:22.896905 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.896856 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" podUID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" containerName="payload-pre-processing" containerID="cri-o://4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3" gracePeriod=30 Sep 18 18:26:22.941908 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:22.941872 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f079112b-b34c-4d57-9b32-a1b4300afd1c" path="/var/lib/kubelet/pods/f079112b-b34c-4d57-9b32-a1b4300afd1c/volumes" Sep 18 18:26:23.045118 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.045050 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f"] Sep 18 18:26:23.047246 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.047220 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-341bd4-2n57f"] Sep 18 18:26:23.173259 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.173234 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:26:23.175783 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.175763 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:26:23.273971 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.273934 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume\") pod \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " Sep 18 18:26:23.274152 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.273998 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhvpb\" (UniqueName: \"kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb\") pod \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " Sep 18 18:26:23.274152 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.274031 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume\") pod \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\" (UID: \"816e406b-7b02-4bf3-b0f7-be41f02c88e5\") " Sep 18 18:26:23.274152 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.274077 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8dmf\" (UniqueName: \"kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf\") pod \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\" (UID: \"00bae033-c422-4dda-bbaf-d4f9f52d8cdb\") " Sep 18 18:26:23.274405 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.274380 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00bae033-c422-4dda-bbaf-d4f9f52d8cdb" (UID: "00bae033-c422-4dda-bbaf-d4f9f52d8cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:26:23.274448 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.274422 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "816e406b-7b02-4bf3-b0f7-be41f02c88e5" (UID: "816e406b-7b02-4bf3-b0f7-be41f02c88e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 18:26:23.276267 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.276246 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf" (OuterVolumeSpecName: "kube-api-access-n8dmf") pod "00bae033-c422-4dda-bbaf-d4f9f52d8cdb" (UID: "00bae033-c422-4dda-bbaf-d4f9f52d8cdb"). InnerVolumeSpecName "kube-api-access-n8dmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:26:23.276386 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.276365 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb" (OuterVolumeSpecName: "kube-api-access-zhvpb") pod "816e406b-7b02-4bf3-b0f7-be41f02c88e5" (UID: "816e406b-7b02-4bf3-b0f7-be41f02c88e5"). InnerVolumeSpecName "kube-api-access-zhvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 18:26:23.375505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.375468 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8dmf\" (UniqueName: \"kubernetes.io/projected/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-kube-api-access-n8dmf\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:23.375505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.375498 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bae033-c422-4dda-bbaf-d4f9f52d8cdb-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:23.375505 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.375508 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvpb\" (UniqueName: \"kubernetes.io/projected/816e406b-7b02-4bf3-b0f7-be41f02c88e5-kube-api-access-zhvpb\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:23.375740 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.375517 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816e406b-7b02-4bf3-b0f7-be41f02c88e5-plugins-config-volume\") on node \"ip-10-0-133-72.ec2.internal\" DevicePath \"\"" Sep 18 18:26:23.832584 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.832548 2570 generic.go:356] "Generic (PLEG): container finished" podID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" containerID="4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3" exitCode=0 Sep 18 18:26:23.832739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.832593 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" Sep 18 18:26:23.832739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.832634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" event={"ID":"816e406b-7b02-4bf3-b0f7-be41f02c88e5","Type":"ContainerDied","Data":"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3"} Sep 18 18:26:23.832739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.832697 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k" event={"ID":"816e406b-7b02-4bf3-b0f7-be41f02c88e5","Type":"ContainerDied","Data":"e5b24c3438cc401dd50211a6898bf330feab14f77d3a3557d83bf7b30aee8c93"} Sep 18 18:26:23.832739 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.832734 2570 scope.go:117] "RemoveContainer" containerID="4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3" Sep 18 18:26:23.834044 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.834020 2570 generic.go:356] "Generic (PLEG): container finished" podID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" containerID="b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13" exitCode=0 Sep 18 18:26:23.834153 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.834051 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" Sep 18 18:26:23.834153 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.834077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" event={"ID":"00bae033-c422-4dda-bbaf-d4f9f52d8cdb","Type":"ContainerDied","Data":"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13"} Sep 18 18:26:23.834153 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.834110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s" event={"ID":"00bae033-c422-4dda-bbaf-d4f9f52d8cdb","Type":"ContainerDied","Data":"ea1094483473f8e7d25211f0b45ef1d7429dd37aead7e076c9e516b510850378"} Sep 18 18:26:23.845548 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.845525 2570 scope.go:117] "RemoveContainer" containerID="4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3" Sep 18 18:26:23.845942 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:26:23.845919 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3\": container with ID starting with 4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3 not found: ID does not exist" containerID="4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3" Sep 18 18:26:23.846029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.845948 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3"} err="failed to get container status \"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3\": rpc error: code = NotFound desc = could not find container \"4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3\": container with ID starting with 4c833834c11885325f3bfe40f5bb4b4a1e14ed442eef4a125b29b49c530204e3 not found: ID does not exist" Sep 18 18:26:23.846029 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.845965 2570 scope.go:117] "RemoveContainer" containerID="b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13" Sep 18 18:26:23.854253 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.854237 2570 scope.go:117] "RemoveContainer" containerID="b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13" Sep 18 18:26:23.854519 ip-10-0-133-72 kubenswrapper[2570]: E0918 18:26:23.854502 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13\": container with ID starting with b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13 not found: ID does not exist" containerID="b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13" Sep 18 18:26:23.854574 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.854526 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13"} err="failed to get container status \"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13\": rpc error: code = NotFound desc = could not find container \"b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13\": container with ID starting with b82a05ff1b7bc3377cf66996f73524f47c04204fec7400b6c2c876ba7e249c13 not found: ID does not exist" Sep 18 18:26:23.861523 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.861502 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:26:23.864891 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.864869 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-341bd4-ccfb4f54b-l825k"] Sep 18 18:26:23.874725 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.874701 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:26:23.878627 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:23.878608 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-341bd4-86496899cc-wxc5s"] Sep 18 18:26:24.941704 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:24.941665 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" path="/var/lib/kubelet/pods/00bae033-c422-4dda-bbaf-d4f9f52d8cdb/volumes" Sep 18 18:26:24.942059 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:24.942023 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" path="/var/lib/kubelet/pods/816e406b-7b02-4bf3-b0f7-be41f02c88e5/volumes" Sep 18 18:26:24.942337 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:24.942323 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e74640-2460-4173-9d97-ee0f94790872" path="/var/lib/kubelet/pods/f8e74640-2460-4173-9d97-ee0f94790872/volumes" Sep 18 18:26:57.519861 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:26:57.519827 2570 scope.go:117] "RemoveContainer" containerID="16022ab15e5d74b080c831f74aa9f83cbfca32afe821182c8c46680961521dc8" Sep 18 18:30:57.576639 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:30:57.576608 2570 scope.go:117] "RemoveContainer" containerID="493cecdac0597cc152c55d41172f6ed9fdf46e1e1cc800ea1532e49c4d4f8c7c" Sep 18 18:30:57.584913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:30:57.584897 2570 scope.go:117] "RemoveContainer" containerID="26e46f2dcd535111278faba4f063d6c5ff6a68fae047cb98c9dfd7bdf8c0613f" Sep 18 18:31:57.606484 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:31:57.606450 2570 scope.go:117] "RemoveContainer" containerID="45be18817f500a2b4b33f70d7f87381d2ceb7d6f67b39fcb53402f2bc5e60b11" Sep 18 18:31:57.615121 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:31:57.615101 2570 scope.go:117] "RemoveContainer" containerID="2eab8c79b2fd71e5a166dea2865178ee28d07195c47ea18943588e78121317bb" Sep 18 18:32:44.421675 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:44.421639 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x5t7d_f23a1124-405d-4273-83a3-cc0abd1a4db8/server/0.log" Sep 18 18:32:44.540249 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:44.540211 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-54g5v_78abb58b-3458-4972-a656-d979f51c3b21/manager/0.log" Sep 18 18:32:44.874044 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:44.873959 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-xw2dc_2f1f88f2-1183-46c2-a013-00aeedc21116/copy-manifests/0.log" Sep 18 18:32:44.891345 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:44.891315 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-xw2dc_2f1f88f2-1183-46c2-a013-00aeedc21116/manager/0.log" Sep 18 18:32:47.412639 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:47.412610 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll_49a091d1-e86e-4198-8507-bffcfa40906d/istio-proxy/0.log" Sep 18 18:32:47.530196 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:47.530159 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bpw6c_124371a9-200d-4575-aeb7-314b79294334/discovery/0.log" Sep 18 18:32:47.854208 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:47.854116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/1.log" Sep 18 18:32:48.705112 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:48.705081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rd2fz_4180bffa-61bf-466d-aefc-d95a87cbfdca/main/0.log" Sep 18 18:32:48.921422 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:48.921391 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5dt8m_0177179c-3cc8-4624-b7f2-66edec58b160/main/0.log" Sep 18 18:32:57.636724 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:57.636692 2570 scope.go:117] "RemoveContainer" containerID="391be43e6581e47c9c2ec9d39d6fb4f82feefb3714a9fb3d05248b0805255deb" Sep 18 18:32:57.713816 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:57.713783 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-58kvg_5958b2d8-31bd-4b8d-8672-024520ce6ef6/global-pull-secret-syncer/0.log" Sep 18 18:32:57.942240 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:57.942209 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p5t9l_d6e1842d-4e29-463e-a581-3e102a65c66b/konnectivity-agent/0.log" Sep 18 18:32:57.999022 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:32:57.998993 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-72.ec2.internal_edf4d681d04ec5f43785f7bccdc8c978/haproxy/0.log" Sep 18 18:33:05.193427 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:05.193397 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9bq6_728290a5-8d19-4b0f-81ba-11f2a6fba8ca/node-exporter/0.log" Sep 18 18:33:05.214647 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:05.214614 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9bq6_728290a5-8d19-4b0f-81ba-11f2a6fba8ca/kube-rbac-proxy/0.log" Sep 18 18:33:05.238991 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:05.238966 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9bq6_728290a5-8d19-4b0f-81ba-11f2a6fba8ca/init-textfile/0.log" Sep 18 18:33:07.139540 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.139496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-g7blk_0c151da0-f847-400f-89c7-7b5fdaca26de/networking-console-plugin/0.log" Sep 18 18:33:07.841563 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841530 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj"] Sep 18 18:33:07.841909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841892 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" containerName="payload-pre-processing" Sep 18 18:33:07.841997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841913 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" containerName="payload-pre-processing" Sep 18 18:33:07.841997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841927 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" containerName="payload-processing" Sep 18 18:33:07.841997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841935 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" containerName="payload-processing" Sep 18 18:33:07.841997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841955 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e74640-2460-4173-9d97-ee0f94790872" containerName="revoke-keys" Sep 18 18:33:07.841997 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.841963 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e74640-2460-4173-9d97-ee0f94790872" containerName="revoke-keys" Sep 18 18:33:07.842282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.842055 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bae033-c422-4dda-bbaf-d4f9f52d8cdb" containerName="payload-processing" Sep 18 18:33:07.842282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.842068 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e74640-2460-4173-9d97-ee0f94790872" containerName="revoke-keys" Sep 18 18:33:07.842282 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.842080 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="816e406b-7b02-4bf3-b0f7-be41f02c88e5" containerName="payload-pre-processing" Sep 18 18:33:07.845902 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.845880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:07.848913 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.848896 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tkl5n\"/\"kube-root-ca.crt\"" Sep 18 18:33:07.849021 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.848917 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tkl5n\"/\"default-dockercfg-btmrw\"" Sep 18 18:33:07.849021 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.848981 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tkl5n\"/\"openshift-service-ca.crt\"" Sep 18 18:33:07.855698 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.855676 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj"] Sep 18 18:33:07.934672 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.934639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-lib-modules\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:07.934849 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.934713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-podres\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:07.934849 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.934735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9wn\" (UniqueName: \"kubernetes.io/projected/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-kube-api-access-7c9wn\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:07.934849 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.934760 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-sys\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:07.934849 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:07.934776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-proc\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035276 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-podres\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035276 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9wn\" (UniqueName: \"kubernetes.io/projected/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-kube-api-access-7c9wn\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035473 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035392 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-podres\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035473 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-sys\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035473 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-proc\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035582 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-lib-modules\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035582 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-proc\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035582 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-sys\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.035704 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.035595 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-lib-modules\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.043909 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.043882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9wn\" (UniqueName: \"kubernetes.io/projected/260a6bd1-cd76-47e6-aeff-ab6095fbd37a-kube-api-access-7c9wn\") pod \"perf-node-gather-daemonset-6pvrj\" (UID: \"260a6bd1-cd76-47e6-aeff-ab6095fbd37a\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.158819 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.158800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.430461 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.430433 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj"] Sep 18 18:33:08.432122 ip-10-0-133-72 kubenswrapper[2570]: W0918 18:33:08.432093 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod260a6bd1_cd76_47e6_aeff_ab6095fbd37a.slice/crio-390f6d542cd44d42ae93a724a4211726b29af0b5aad10e2ae626bc94b160411a WatchSource:0}: Error finding container 390f6d542cd44d42ae93a724a4211726b29af0b5aad10e2ae626bc94b160411a: Status 404 returned error can't find the container with id 390f6d542cd44d42ae93a724a4211726b29af0b5aad10e2ae626bc94b160411a Sep 18 18:33:08.829357 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.829260 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" event={"ID":"260a6bd1-cd76-47e6-aeff-ab6095fbd37a","Type":"ContainerStarted","Data":"9abf565f2e7e14d23dc22224ae8af4996311ddc1e77654bdb96ea2ef83adc0e3"} Sep 18 18:33:08.829357 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.829323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" event={"ID":"260a6bd1-cd76-47e6-aeff-ab6095fbd37a","Type":"ContainerStarted","Data":"390f6d542cd44d42ae93a724a4211726b29af0b5aad10e2ae626bc94b160411a"} Sep 18 18:33:08.829546 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.829366 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:08.848524 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:08.848480 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" podStartSLOduration=1.8484655939999999 podStartE2EDuration="1.848465594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:33:08.847235585 +0000 UTC m=+2593.939890693" watchObservedRunningTime="2026-09-18 18:33:08.848465594 +0000 UTC m=+2593.941120681" Sep 18 18:33:10.283333 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:10.283262 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ll45_9172cf61-f370-4720-9191-b5ea2f68d55e/dns/0.log" Sep 18 18:33:10.311970 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:10.311947 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ll45_9172cf61-f370-4720-9191-b5ea2f68d55e/kube-rbac-proxy/0.log" Sep 18 18:33:10.521734 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:10.521702 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j6k5v_717ab396-8f0c-45aa-bd46-6d524b982255/dns-node-resolver/0.log" Sep 18 18:33:11.249280 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:11.249256 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g6r9r_a88fbff8-83e6-46e6-8126-87ab6c7d9a7b/node-ca/0.log" Sep 18 18:33:12.305167 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:12.305132 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll_49a091d1-e86e-4198-8507-bffcfa40906d/istio-proxy/0.log" Sep 18 18:33:13.095772 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:13.095747 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bpw6c_124371a9-200d-4575-aeb7-314b79294334/discovery/0.log" Sep 18 18:33:13.229418 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:13.229390 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/0.log" Sep 18 18:33:13.233981 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:13.233960 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/1.log" Sep 18 18:33:14.290438 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:14.290403 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmlnt_683adc43-c28a-4c5f-a783-88a97ff26e30/serve-healthcheck-canary/0.log" Sep 18 18:33:14.847180 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:14.847152 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-6pvrj" Sep 18 18:33:14.989958 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:14.989925 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ds66_53ad7cfa-75ac-4f0a-b524-84def11fb4ce/kube-rbac-proxy/0.log" Sep 18 18:33:15.037754 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:15.037728 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ds66_53ad7cfa-75ac-4f0a-b524-84def11fb4ce/exporter/0.log" Sep 18 18:33:15.096807 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:15.096781 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ds66_53ad7cfa-75ac-4f0a-b524-84def11fb4ce/extractor/0.log" Sep 18 18:33:18.392789 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:18.392761 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll_49a091d1-e86e-4198-8507-bffcfa40906d/istio-proxy/0.log" Sep 18 18:33:18.767927 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:18.767898 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x5t7d_f23a1124-405d-4273-83a3-cc0abd1a4db8/server/0.log" Sep 18 18:33:18.890794 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:18.890765 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bpw6c_124371a9-200d-4575-aeb7-314b79294334/discovery/0.log" Sep 18 18:33:19.067544 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:19.067470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-54g5v_78abb58b-3458-4972-a656-d979f51c3b21/manager/0.log" Sep 18 18:33:19.106439 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:19.106411 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/0.log" Sep 18 18:33:19.111592 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:19.111566 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/1.log" Sep 18 18:33:19.463991 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:19.463957 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-xw2dc_2f1f88f2-1183-46c2-a013-00aeedc21116/manager/0.log" Sep 18 18:33:19.533484 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:19.533457 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-xw2dc_2f1f88f2-1183-46c2-a013-00aeedc21116/copy-manifests/0.log" Sep 18 18:33:20.846106 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:20.846065 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-9h7ll_49a091d1-e86e-4198-8507-bffcfa40906d/istio-proxy/0.log" Sep 18 18:33:21.088691 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:21.088655 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-786dfb84b9-ps597_669eecf6-1b5f-4b08-9b30-b40a64fb27ce/manager/0.log" Sep 18 18:33:21.508464 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:21.508429 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bpw6c_124371a9-200d-4575-aeb7-314b79294334/discovery/0.log" Sep 18 18:33:21.678557 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:21.678526 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/0.log" Sep 18 18:33:21.683780 ip-10-0-133-72 kubenswrapper[2570]: I0918 18:33:21.683758 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-62zx7_39ca5410-7955-4a3d-8890-f90cb83ce783/istio-proxy/1.log"